Show HN: DeepSeek-Coder-V2-Lite Under 5GB: 65 TPS 100% In-VRAM on 6GB GPU (huggingface.co)

Aman Sachan has announced the DeepSeek-Coder-V2-Lite, utilizing a novel technique called Manifold-Activated Pruning of Latent Experts (MAPLE) that allows for a 16 billion parameter model to be compressed to just 4.50 GB. This model can fully execute in VRAM on a 6GB NVIDIA GeForce RTX 4050 laptop GPU, achieving an impressive inference speed of 65 to 76 tokens per second. Traditional implementations faced significant limitations due to excessive hardware requirements, often necessitating hybrid CPU/GPU offloading that drastically reduced throughput. The significance of this development lies in its ability to democratize access to powerful AI models by making them amenable to consumer-grade hardware. By retaining 84.4% of the core routing experts while shrinking the model size, MAPLE not only enhances efficiency but also maintains performance quality. This achievement opens the door for broader adoption of AI-powered applications where GPU memory constraints were previously a barrier, thus paving the way for innovative solutions that can run on more accessible devices without compromising capability.