From GPT2 to Kimi3 (twitter.com)

A recent development in AI/ML research has unveiled the Kimi3 model, which boasts a staggering scale of 2.8 trillion parameters—equivalent to 22,580 GPT-2 models combined. This leap in capacity marks a significant milestone in the field, demonstrating a dramatic increase in model size and capabilities within just seven years. The announcement emphasizes the importance not only of scaling but also of architectural advancements leading to this unprecedented capability. Kimi3 utilizes a decoder-only transformer architecture, refining the process of token handling through an innovative key-value (KV) caching mechanism. This approach alleviates inefficiencies typically seen in autoregressive decoding, where only the logits for the final token position are necessary, allowing the model to store and reuse projections for prior inputs. Furthermore, the introduction of linear attention enhances scalability by modifying the traditional softmax attention mechanism, enabling the model to handle larger datasets more efficiently. The implications of Kimi3 extend beyond mere size, pushing the boundaries of performance and efficiency in natural language processing applications.