Kimi K3, and what we can still learn from the pelican benchmark (simonwillison.net)

Moonshot AI has unveiled Kimi K3, their most powerful AI model to date, boasting 2.8 trillion parameters. This launch positions K3 as the first "open 3T-class model," overtaking DeepSeek’s previous 1.6 trillion parameter model. Initial benchmarks suggest K3 outperforms popular models like Claude Opus 4.8 and GPT-5.5 but lags slightly behind Claude Fable 5 and GPT-5.6 Sol. The model is available via an API, with a full weight release expected by July 27, 2026. K3 features a competitive pricing strategy at $3 per million input tokens and $15 for output tokens, marking it as the most expensive offering from a Chinese AI lab thus far. The significance of this release lies not only in its scale and pricing but also in its potential implications for the AI/ML community. The reduced token usage—21% less than its predecessor—indicates optimizations in processing. Additionally, K3's performance on various benchmarks, including a notable presence on Arena.ai’s coding tasks, highlights its versatility. However, the unconventional pelican benchmark utilized to test K3 serves as a reminder of the limitations in assessing model capabilities, particularly in complex reasoning tasks. This showcases the ongoing evolution in evaluating AI technologies and underscores the need for more relevant benchmarks that reflect real-world applications.