Researchers propose Hierarchical Landmark Sparse (HiLS) Attention, a chunk-wise sparse mechanism that learns chunk selection end-to-end under the language-modeling loss to address the quadratic cost and poor length extrapolation of dense attention.
- HiLS factorizes attention hierarchically, allowing each query to extract chunk-specific information independently before fusing outputs based on retrieval scores.
- The method optimizes retrieval scores directly with the LM loss, enabling native sparse training without inaccurate chunk selection.
- HiLS-Attention extrapolates more than 64 times the training context length with 90% retrieval accuracy, outperforming full attention in long-context scenarios.
- Existing full-attention models can be converted to HiLS-Attention via lightweight continued pretraining, preserving in-domain performance while gaining ultra-long-context capabilities.
HiLS-Attention breaks the usual efficiency-performance trade-off by combining sparse KV access with superior extrapolation, enabling long-context LLMs that are both more efficient and effective than their full-attention counterparts.