Unleashing the Potential of Imperfect Demonstration for Imitation Learning via Hierarchical Expert Guidance.

Abstract

Despite the substantial progress of imitation learning (IL) in training agents to mimic expert behavior, existing methods still suffer from covariate shift and compounding errors due to the limited availability of expert demonstrations and imperfect suboptimal trajectories. In this article, we propose hierarchical expert guidance imitation learning (HEED), a two-stage framework that aims to transform imperfect trajectories into reliable expert demonstrations. In particular, our hierarchical expert guidance enables distribution-level correction and transition-level selection to expand the effective expert distribution. To achieve this, our expert guidance first acts at the correction level, which enables those imperfect trajectories to become reliable candidates. Then, we derive a discriminator-guided proximity function with the expert guidance at the selection level to allow selecting transitions statistically close to the expert distribution, thus filtering out those less imperfect ones. Notably, we also provide the theoretical analysis that the correction process could reduce divergence between expert and suboptimal distributions, while the selection mechanism further bounds policy suboptimality. The extensive empirical results in diverse D4RL benchmarks show that our method consistently outperforms existing baselines, demonstrating strong generalization even under extremely limited and noisy expert supervision.