Qwen team introduces HydraHead, a head-level hybrid attention architecture

The Qwen team has introduced HydraHead, a novel neural network architecture that hybridizes Full Attention (FA) and Linear Attention (LA) along the head axis to address the quadratic complexity bottleneck of long-context processing.

• The design leverages head-level functional heterogeneity, using an interpretability-driven selection strategy to preserve FA only for retrieval-critical heads.
• A scale-normalized fusion module reconciles the distributional gap between FA and LA head outputs.
• Trained on only 15B tokens, HydraHead achieves over 69% improvement over the baseline at a 512K context length.
• It matches a 3:1 layer-wise hybrid's long-context performance at a 7:1 LA-to-FA ratio while maintaining strong general reasoning capabilities.

This approach highlights the significant scaling potential of head-level hybridization, allowing the model to approach the performance of Qwen3.5, which has a native context length of 256K.