Researchers propose a unified framework for diagnosing why vision-language models fail by distinguishing between visual recognition bottlenecks and knowledge retrieval deficits. The study investigates whether pre-generation signals can predict these specific failure sources across various datasets and model families.
- Failures arising from visual or recognition bottlenecks are best captured by visual-token representations.
- Errors persisting after entity identification are better predicted by prompt-conditioned hidden states.
- These pre-generation signals enable efficient failure-source prediction before the model produces an answer.
- Uncertain cases can be routed to targeted interventions such as image repair, entity recognition support, or external retrieval.
This approach allows for more precise diagnosis of uncertainty in VLMs, enabling specific corrective actions rather than treating incorrect answers as monolithic failures.