How I use GBNF grammars to make small local models stop hallucinating JSON (eris-system.dev)

In a recent post, a developer detailed their innovative use of GBNF (Generalized Backus-Naur Form) grammars to enhance the reliability of small local AI models in generating valid JSON outputs. The project revolves around "Eris," a local-first AI agent built in Rust that employs 8-26B models from llama.cpp. The major engineering hurdle was ensuring these models consistently produce schema-conformant JSON for tool-calling protocols, as they often exhibited unpredictable behaviors such as omitting brackets or generating incorrect keys. The author demonstrates that leveraging GBNF grammars allows for strict enforcement of JSON structures at the token level, fundamentally preventing models from outputting malformed JSON. This advancement is significant for the AI/ML community, especially for developers working with local models where server-side APIs are unavailable. By compiling GBNF rules that specify the exact JSON structure required, including dynamic rules for tool arguments based on predefined schemas, the model can be constrained to produce only valid outputs. Additionally, the approach involves real-time narrowing of available grammar based on the user's query, which reduces potential errors. This technique not only addresses common pitfalls in local AI implementations but also showcases a scalable method for ensuring compliance with structured output formats, paving the way for more robust local AI applications.