Show HN: DeepSeek V3 in ~200 lines of pure Jax (github.com)

A new AI architecture, DeepSeek V3, has been announced, implemented in just 190 lines of pure JAX without relying on frameworks like Flax or PyTorch. This model was pretrained from scratch on a collection of public-domain detective fiction, demonstrating a novel approach to latent attention and mixture of experts (MoE). Among its key features is the Multi-head Latent Attention (MLA), which reduces the necessary computations per token, caching just a small latent instead of full keys and values, and an auxiliary loss-balancing technique that optimally manages expert utilization without impacting performance. The significance of DeepSeek V3 lies in its compact design that enhances clarity and simplicity while maintaining robust performance. With a total of 36.2 million parameters and an impressive median training speed of about 10.7k tokens per second on a MacBook Air, the model achieves a validation loss reduction from 5.02 to 3.51 over training. Importantly, it offers an exploratory perspective on AI architecture, serving as both a demonstration of innovative techniques in sparse routing and a platform for future enhancements. The release not only includes functional code but also emphasizes reproducibility, enabling further research within the AI/ML community.