Llama-modchip: 284B DeepSeek on 64GB RAM, no GPU (github.com)

A groundbreaking development in AI computing, the Llama-modchip enables the execution of the 284B DeepSeek-V4-Flash model on standard CPUs with just 64GB of RAM—eliminating the need for any GPU. This marks a significant leap as it allows both mainstream developers and researchers to perform complex deep learning tasks previously reserved for data centers. Notably, the model fits entirely in RAM, achieving an impressive decoder speed of up to 111.6 tokens per second for 30B MoEs, ensuring high performance without the expense of specialized hardware. Key technical advancements include the implementation of the Analytical SwiGLU-H2 compression technique and Fractal-64 architecture, which collectively reduce the model size from 76.87 GiB to 56.56 GiB, facilitating real-time inference capabilities on typical CPU workstations. The modchip also introduces rapid warm restores for interactive tools, further streamlining user experience by reducing latency in computational tasks. By democratizing access to advanced AI models, this development holds the potential to accelerate innovation across various fields that leverage AI/ML technologies.