Tencent has released the HiLS-Attention-7B checkpoint, a 7-billion parameter model built on an OLMo3-style backbone. The model implements a chunk-wise sparse attention mechanism that learns chunk selection end-to-end under the language-modeling loss.
The architecture uses compressed chunk keys to estimate chunk mass and factorizes attention into inter-chunk and intra-chunk softmax, enabling native sparse training for efficient long-context modeling. The checkpoint is continued-trained from the base OLMo3-7B weights and is available via the Tencent-Hunyuan repository. This is a pretrained base model without alignment or safety tuning, meaning it may reflect biases present in the training corpus.