Hidden Decoding scales LLMs by expanding tokens into streams

Researchers propose Hidden Decoding, a sequence-length scaling method applied during continued pretraining that allows existing Transformer backbones to improve without costly retraining or architectural changes. The approach expands each token into n streams with independent embedding tables and uses Stream-Factorized Attention to reduce computational costs from quadratic to roughly linear in n.

• WeLM-HD4-80B and WeLM-HD4-617B models were trained at n=4, improving upon matched non-HD baselines.
• Hidden Decoding is the first demonstrated sequence-length scaling method at the 100B+ MoE scale.
• Performance gains increase as the expansion factor n increases, confirming it as a practical fixed-backbone scaling path for frontier-scale LLMs.