This article outlines a specific sequence of research papers and model reports to help readers comprehend the technical evolution behind Moonshot AI's Kimi K3 model. It argues that K3 is not an isolated release but the convergence of several distinct research threads, including linear attention mechanisms and sparse Mixture-of-Experts designs.
- Linear Transformers Are Secretly Fast Weight Programmers: Establishes the foundation by interpreting linear attention as a system updating associative memory.
- Gated DeltaNet (arXiv:2412.06464): Introduces gated delta updates for improving state management over long sequences.
- Kimi Linear / Kimi Delta Attention (KDA): Presents the hybrid linear-attention architecture that serves as K3's backbone.
- LatentMoE and Stable LatentMoE: Details the latent-space routing formulation and its evolution, which allows K3 to activate 16 out of 896 routed experts per token.
- Attention Residuals (arXiv:2603.15031): Explains the mechanism allowing the model to decide how to use residual networks for better information flow.
- Kimi Model Evolution: Traces the progression from Kimi K1.5 through K2.5 to K3, showing how each iteration refined training recipes and infrastructure.
Following this path provides the necessary context to understand why specific architectural choices were made in Kimi K3 rather than viewing it as a standalone product.