Laguna XS 2.1 33B on a RTX 3090: 296 tok/s peak, 152 tok/s at 256K context (www.lucebox.com)

In a significant advancement for the AI and machine learning community, poolside's Laguna XS 2.1 model, when deployed on an NVIDIA RTX 3090, achieves impressive decoding speeds of up to 296 tokens per second (tok/s) for short contexts, while maintaining 152 tok/s even at a staggering 256K tokens. This performance leap is attributed to three key optimizations: the introduction of a context-KV ring cache, the use of sliding-window ring caches, and KVFlash paging, which allows for effective processing of contexts that exceed the GPU's physical memory limits. Notably, the speculative decoding mechanism maintains output quality, ensuring that every token generated is consistent with the model's capabilities. The implications of these enhancements are profound, showcasing that efficient utilization of GPU memory and advanced decoding techniques can drastically improve the processing of large-scale contexts—historically a bottleneck in natural language processing tasks. With prefill speeds now reaching an impressive ~3,500 tok/s for 256K tokens, these optimizations not only enhance throughput but also reduce the time for context preparation, which was previously a major constraint. As such, this development positions Laguna XS 2.1 as a flagship example of how cutting-edge optimizations can redefine practical applications in AI, enabling complex tasks to be executed more efficiently and accurately.