Structured Sparse AutoEncoder enforces concept consistency across modalities

Researchers propose a Structured Sparse AutoEncoder (S²AE) that addresses the struggle of vanilla SAEs to learn modality-consistent concepts in vision-language models. The method groups image patches based on Transformer attention similarity and spatial proximity, applying structured sparsity regularization to enforce semantic and spatial consistency.

• Evaluated on Qwen2.5-VL-7B-Instruct, S²AE achieves a 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm).
• The approach maintains near-perfect reconstruction fidelity with an Explained Variance above 99%.
• Cross-modal analysis shows a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities.

By driving latent neurons to specialize in distinct, semantically grounded concepts, S²AE fosters more coherent and disentangled representations across visual and textual inputs.