We self-host DeepSeek V4 Flash on AWS spot instances (twitter.com)

Vercantez has launched a self-hosted free tier of DeepSeek V4 Flash powered by AWS spot instances, aiming to offer a sustainable service without the financial drawbacks associated with traditional free tiers. By utilizing a mixture-of-experts model with 284 billion total parameters but only 13 billion active per token, the company is able to maintain performance while keeping costs manageable. This setup is optimized for concurrent usage, allowing for efficient resource allocation, especially through advanced techniques like speculative decoding that improves response times by predicting multiple tokens at once. The significance of this development for the AI/ML community lies in its innovative approach to balancing performance demands with cost management through self-hosting and open-source solutions. The model's quantization into the 4-bit MXFP4 format not only reduces memory usage significantly—making it feasible to run on NVIDIA's RTX PRO 6000 GPUs—but it also showcases the efficiency of combining advanced AI with cloud services to meet high user demand without compromising on quality. The successful implementation of techniques such as caching and fair scheduling positions this setup as a potential blueprint for startups and developers seeking to leverage AI models cost-effectively.