Production-Ready Speculative Decoding Models and Framework (lmsys.org)

The SpecForge team has introduced SpecBundle (Phase 1), a set of production-ready EAGLE-3 model checkpoints, in collaboration with industry partners like Ant and Nex-AGI. This initiative aims to enhance the practical implementation of speculative decoding, a technique that accelerates large language model (LLM) inference by using a draft model to propose tokens verified by a more powerful target model. SpecBundle improves performance and availability by training on extensive datasets, thus addressing the existing challenges like limited draft models and suboptimal open-source tooling. Significantly, the introduction of SpecForge v0.2 comes with substantial usability improvements, including multi-backend support and optimized data processing, which collectively enhance the framework's scalability and maintainability. The initiative offers various advantages, such as reduced inference costs and increased efficiency in reinforcement learning workflows. By equipping mainstream open-source models with high-performance draft weights and expanding the range of instruct-tuned models, SpecBundle promises to democratize speculative decoding and promote broader adoption across the AI community, facilitating faster and more cost-effective local and enterprise model deployments.