DeepSeek Ultrafast (github.com)

DeepSeek has unveiled an ultra-fast inference engine, DeepSeek-V4.1-Flash, designed for high-performance AI applications, specifically optimized for sm_100a and sm_103a GPU architectures. This engine boasts impressive capabilities, such as supporting up to 1 million tokens for context length and enabling multi-server key-value (KV) cache sharing. The performance metrics are staggering, with decoding speeds of 1,740 tokens per second on a 4 GPU setup and 2,060 tokens per second with 8 GPUs. Such capabilities allow for complex tasks, like generating HTML code for a Tetris game in real-time. This launch is significant for the AI/ML community because it accelerates inference speeds dramatically while improving load balancing across server resources, thanks to its innovative use of Redis for cache management and RDMA for efficient data transfer. The DeepSeek engine's unique cache compression mechanisms minimize latency and facilitate rapid access to conversation histories, making it a robust solution for applications requiring quick, responsive AI interactions. Furthermore, the engine supports open-source deployment through Baseten's Model APIs, promising additional performance enhancements in future releases.