A study investigates how visual information is routed within vision-language models (VLMs) using causal patching on synthetic and natural datasets. The research identifies two primary mechanisms: a direct pathway where visual data remains in image tokens, and a text-mediated pathway where information transfers to query tokens first.
- Models utilize a direct pathway by retaining visual information in image token representations for later readout.
- A text-mediated pathway involves transferring visual data to query tokens before the final token processes it.
- Pathway selection depends on the specific task, data distribution, and prompt design.
- Attention knockouts and corrupted-input patching demonstrate that models can flexibly switch to the text-mediated pathway as a fallback when the primary route is ablated.
These findings provide a mechanistic characterization of visual information flow in VLMs and highlight the flexibility of their internal mechanisms under intervention.