Toolcall-doctor – Shrink broken LLM tool-call reproducers (github.com)

The newly announced toolcall-doctor simplifies the process of debugging and minimizing requests that reproduce tool-calling bugs in large language models (LLMs). By automating the iterative process of removing parts of the request and rerunning the model to verify that the specified failure still occurs, this tool can significantly reduce the size of requests — achieving compressions of up to 68.27%. This not only streamlines the debugging workflow for developers but also ensures that essential properties known as "keepers" are preserved, thus providing a reliable means to isolate issues in complex interactions with AI models. The significance of toolcall-doctor lies in its potential to enhance productivity in the AI/ML community by reducing the time and effort required to troubleshoot model failures. The tool employs a character-level delta debugging approach, allowing users to specify failure conditions and essential attributes to maintain during minimization. Developers can run this tool on locally validated models like Ollama and LLaMA, making it a robust resource for anyone looking to refine their interactions with LLMs effectively. This automation could lead to more efficient identification and resolution of bugs, ultimately advancing the reliability of AI applications.