Sparse transformers have more interpretable individual parameters than dense ones

Researchers investigated whether individual weights in neural networks can be understood globally across the full training distribution by characterizing when they matter. They introduced an automated LLM pipeline that generates and verifies human-readable descriptions of weight functions on held-out text.

• The study compared two sparse and two dense transformers to measure the fraction of interpretable weights.
• A meaningful fraction of a sparse transformer model's weights can be interpreted, with 12 to 31% having a single short description identifying their use.
• The gap in interpretability between sparse and dense models widens when unreliable descriptions are discarded.

The results demonstrate that individual parameters in sparse transformers are more amenable to mechanistic interpretation than those in dense architectures.