Analysis-driven Transformer linearization reduces approximation errors in LLaMA and Qwen

This work addresses the quadratic cost of causal self-attention by isolating the effect of state update design in a strict frozen-backbone regime. The authors show that softmax relies on key-dependent, rank-1 orthogonal projections, explaining why delta-style networks outperform purely gated accumulation.

• Identifies a potential source of approximation errors and introduces structural interventions including sink tokens, short convolutions, and fixed-budget cache routing.
• Scales the linearization approach across LLaMA and Qwen models up to 32B parameters.
• Outperforms prior post hoc baselines on MMLU and matches the long-context retrieval of complex adaptive-caching frameworks.

The study provides a method to reduce the performance gap in linearized transformers while maintaining efficiency.