Structured Sparse AutoEncoder enforces concept consistency across modalities

Researchers propose a Structured Sparse AutoEncoder (S²AE) that enforces concept consistency from both semantic and spatial perspectives in the visual modality of vision-language models. The method groups image patches based on Transformer attention similarity and spatial proximity, introducing structured sparsity regularization to drive latent neurons toward distinct, semantically grounded concepts.

• Evaluated on Qwen2.5-VL-7B-Instruct, S²AE achieves a 6.06% average improvement in semantic alignment (mIoU) and 60.81 in representational efficiency (lower l0 norm).
• The approach maintains near-perfect reconstruction fidelity with an Explained Variance above 99%.
• Cross-modal analysis shows a 3.08% average gain in semantic consistency and a 2.37% average gain in monosemanticity scores for both modalities of multimodal features.

By leveraging visual structural priors, S²AE enhances neuronal monosemanticity, fostering more coherent and disentangled representations across modalities.