Researchers propose a meta-learning framework for Reinforcement Learning from Human Feedback and Direct Preference Optimization to address the challenge of unequal human preference data availability across languages. By leveraging preference data from other languages, the method learns a transferable initialization that allows effective adaptation to a target language using very few samples.
- The approach provides theoretical guarantees for both meta-reward modeling and meta-policy optimization settings.
- In extremely low-resource settings with only 100 target-language preference samples, it achieves up to 28% win-rate improvements over baseline methods.
- The method consistently outperforms baselines across multiple target languages and model scales.
- Advantages are retained across different combinations of meta-training languages and varying linguistic distances from the target languages.
This approach helps mitigate data scarcity issues in multilingual alignment by enabling effective adaptation with minimal labeled data.