Never-Ending Behavior-Cloning Agent for Robotic Manipulation.

Abstract

Relying on multimodal observations, embodied robots (e.g., humanoid robots) could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents in robots still face existing long-standing challenges, i.e., 3-D scene representation and human-level task learning, when adapting to a series of new tasks in practical scenarios. We here investigate the above challenges with a never-ending behavior-cloning agent (NBAgent) in embodied robots, a pioneering language-conditioned NBAgent, which can continually learn observation knowledge of novel 3-D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-shared semantic rendering module (SSR) and a skill-shared representation distillation module (SRD) to effectively learn 3-D scene semantics from skill-shared attributes, further tackling 3-D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner (SEP) to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like humans from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method.