Selective Test-Time Debiasing for CLIP via Reward Gating

Researchers introduce Reward-Gated Test-Time Adaptation (RG-TTA), a reinforcement learning-based framework that selectively applies debiasing to Vision Language Models based on input bias sensitivity. This approach resolves the fairness-utility trade-off inherent in uniform correction methods by adapting regularization only for bias-sensitive queries.

• RG-TTA adaptively triggers fairness regularization during test-time policy adaptation based on the bias sensitivity of each input.
• The framework focuses exclusively on optimizing cross-modal alignment for bias-insensitive inputs to preserve semantic information.
• Experiments on FairFace and UTKFace benchmarks demonstrate substantial bias reduction while simultaneously improving zero-shot utility.