XGrammar-2: 80x Faster Structured Generation for Agent Tool Calling (blog.mlc.ai)

XGrammar-2 has been launched as a significant upgrade to its predecessor, XGrammar, specifically designed for complex agent applications requiring structured output generation. This new version introduces Structural Tag, a JSON-based protocol that standardizes diverse output formats, including the OpenAI harmony format and custom tool-calling structures. With multiple efficiency enhancements—such as cross-grammar caching, repetition-state compression, and support for batching and speculative decoding—XGrammar-2 achieves up to 80 times faster processing speeds, making it suitable for demanding applications that require accurate and varied responses from large language models (LLMs). The upgrade addresses the growing complexity of agent applications by enabling seamless interaction with advanced harnesses and reducing overhead associated with processing extensive structures. As agent models become increasingly sophisticated, the adoption of XGrammar-2 by leading AI labs and companies underscores its importance in ensuring that model outputs conform to required formats without sacrificing accuracy. Key optimizations ensure that even large grammars maintain minimal overhead during generation, which is crucial for real-time or on-device applications. Overall, XGrammar-2 signifies a robust advancement in structured generation for AI systems, enhancing the capabilities of LLMs in delivering reliable, complex outputs efficiently.