Learning generalizable agents via self-supervised exploration.

Abstract

Generalization remains a key challenge in visual reinforcement learning, agents trained in limited views often struggle to generalize the learned skills well to unseen environments. Despite the remarkable progress achieved by self-supervised learning, naively linking self-supervised learning to visual reinforcement learning algorithms may degrade the generalization performance, suffering from lower sample efficiency and unstable training. This paper proposes a novel self-supervised exploration framework for learning the dynamics-relevant representation, which better integrates the representation learning into the reinforcement learning decision-making process. Specifically, our framework consists of two core modules: visual discrepancy inference module (VDIM) and exploration via distributional discrepancy module (EDDM). VDIM ensures sufficient task-relevant information by learning features shared across different views, and filtering out information without predictive power. Designed EDDM to identify changed features through actively exploring the environment, thus boosting the agent's self-awareness of which pixels are beneficial for decision-making and quickly adapting to new scenarios. Extensive experiments demonstrate that our method significantly outperforms prior methods and achieves salient improvements on the generalization capability and sample efficiency.