The authors address the challenge of inverse reinforcement learning (IRL) by introducing a hierarchical machine teaching algorithm that operates across multiple Markov Decision Processes (MDPs). This approach aims to infer reward functions that generalize well across diverse operational contexts rather than overfitting to single environments.
- Theoretical analysis shows that in the unlimited-data regime, comparisons impose strictly stronger global constraints on rewards than other feedback modalities.
- The algorithm greedily selects informative environments to expose complementary reward constraints and strategically queries low-cost feedback within them.
- Empirical results demonstrate substantially lower regret and stronger generalization to held-out environments compared to uniform teaching baselines under identical feedback budgets.
The work highlights the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions in autonomous agents.