Moonshot AI has published details regarding the architecture of its new Kimi K3 model, which offers a radically new approach to handling ultra-long contexts through a hybrid attention mechanism scheme.
What Happened
In a new technical report, Moonshot AI describes the Kimi K3 architecture, which consists of 93 layers. The model uses a unique combination of blocks, including 3 KDA layers and one global MLA layer. Additionally, the architecture implements an Attention Residuals mechanism for efficient information extraction from early blocks and an updated Mixture-of-Experts (MoE) structure, where the number of experts has been increased to 896 (compared to 384 in K2). To optimize training and load balancing, SiTU-GLU activation functions and Quantile Balancing have been introduced.
Context
Traditional models face the problem of linear KV cache bloat when processing long sequences, making work with massive contexts extremely resource-intensive. The previous iteration, K2, used a different structure, whereas K3 focuses on a hybrid approach (KDA + MLA) and MoE scalability to solve memory efficiency and inference challenges.
Why It Matters for the Industry
The shift from monolithic attention mechanisms to hybrid KDA and MLA schemes sets a new direction in the industry, offering an alternative to standard RoPE scaling methods. The use of Attention Residuals and MoE optimization with a larger number of experts allows for the creation of more scalable models capable of efficiently managing memory resources when working with extremely long contexts.
Why It Matters for Users
For end users and application developers, this means the ability to work with colossal amounts of data—up to one million tokens—without critical performance slowdowns or sharp increases in memory consumption. This paves the way for advanced AI agents capable of analyzing entire libraries of books, massive codebases, or complex legal archives within a single session.
Sources
Author
Look at AI, Editorial Staff