RITA robustifies vision-language models via test-time prompt adaptation

The authors propose RITA, a framework that enhances the adversarial robustness of pre-trained Vision-Language Models by shifting from sample-level estimates to distribution-level alignment during test time.

• RITA uses optimal transport to align augmented visual features with textual prototypes, mitigating adversarial outliers and correcting cross-modal semantic misalignment.
• A dynamic cache progressively accumulates reliable cues from the test stream for online refinement.
• Experiments show that RITA significantly improves adversarial robustness without compromising clean accuracy.