Alleged Mythos Class Anthropic Checkpoint Leak (huggingface.co)

A significant leak from Anthropic has revealed details about their upcoming Mythos-class AI model, internally codenamed "numbat." With a staggering architecture featuring approximately 14 trillion parameters, 114 layers, and a hybrid of Gated Delta Networks (GDN) and Multi-head Latent Attention (MLA), this model utilizes a Mixture-of-Experts approach with 1,224 total experts. The leaked checkpoint, pulled from a mid-pretraining stage with around 1.96 million steps anticipated in total, provides developers with a complete implementation and reveals internal workings, including a detailed tensor structure across 772 shards totaling roughly 7.72 TB of data. This leak is pivotal for the AI/ML community, as it offers unprecedented insights into the architecture and configuration of next-gen AI models. The presence of a detailed model reference implementation allows other researchers and developers to explore and potentially build upon Anthropic's advancements. Moreover, the checkpoint's use of NVFP4 quantization for evaluations signals a continued trend towards efficiency in model training and evaluation, emphasizing the importance of scalable, high-performance training environments. This information not only fuels ongoing discussions about the future capabilities of large language models but also underscores the importance of safeguarding sensitive AI development data.