Researchers propose two strategies to improve the feedback efficiency of reinforcement learning from human feedback (RLHF) for diffusion models: selective timestep weighting and advantage-based replay. These methods address the high cost of human or reward model evaluations by emphasizing informative timesteps and trajectories during optimization.
- A per-timestep weighting scheme reweights denoising steps, theoretically connecting to proximal policy optimization (PPO) convergence properties.
- An advantage-based replay mechanism prioritizes informative trajectories, allowing the model to reuse past samples instead of querying new rewards.
- The approach achieves up to a 6x improvement in sample efficiency compared to widely used diffusion RLHF baselines under identical hyperparameter settings.
These strategies significantly enhance the practicality of diffusion RLHF by reducing reliance on extensive feedback while preserving generalization to unseen prompts.