Multi-modal, multi-environment machine teaching improves robust reward learning

The authors address the challenge of inverse reinforcement learning (IRL) by introducing a hierarchical machine teaching algorithm that operates across multiple Markov Decision Processes (MDPs). This approach aims to infer reward functions that generalize effectively despite heterogeneous feedback modalities and varying environment dynamics.

• Theoretical analysis shows that in the unlimited-data regime, comparisons impose strictly stronger global constraints on rewards than other feedback modalities.
• The proposed algorithm greedily selects informative environments to expose complementary reward constraints before querying low-cost feedback within them.
• Empirical results demonstrate substantially lower regret and stronger generalization to held-out environments compared to uniform teaching baselines under identical feedback budgets.

The authors consider this significant because it demonstrates the importance of multi-environment, multi-modal teaching for learning dynamics-robust reward functions that align with human intent across diverse operational contexts.