Visual Semantic Entropy establishes new SOTA for VLM uncertainty estimation

Researchers propose Visual Semantic Entropy (VSE), a method that measures uncertainty in vision-language models by perturbing only the image while keeping the text query fixed. This approach addresses the failure of existing entropy-based methods to accurately capture visual ambiguity due to overconfident embeddings or textual dominance.

• VSE clusters generated answers into semantic prototypes and computes mass-weighted dispersion among them.
• The method isolates visual evidence from prompt sensitivity by avoiding textual paraphrasing.
• Evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates effectiveness.
• VSE establishes a new state-of-the-art for VLM uncertainty estimation.

The authors consider this important because it provides a more accurate measure of visual ambiguity, overcoming the limitations of previous techniques that relied on output diversity or input perturbations.