Speculative Programmatic Tool Calling (alexzhang13.github.io)

A new concept known as Speculative Programmatic Tool Calling (sPTC) has been introduced, allowing for overlapping tool call computations with generated code within a REPL (Read-Eval-Print Loop) environment. This innovative technique draws inspiration from speculative execution in CPUs and aims to improve efficiency by pre-launching tool calls from partially generated code, rather than waiting for complete generation before executing any tool. This is particularly beneficial for incorporating sub-LLMs or agent calls, effectively caching their outputs to mitigate latency issues common in high-latency tool calls. The significance of sPTC lies in its potential to enhance inference time for Recursive Language Models (RLMs) and other harnesses that depend on code execution. By allowing tool calls to be speculated and executed ahead of time, developers can significantly reduce latency. Early experiments indicate that this method can yield runtime improvements in the range of 1-1.2 times. The design philosophy emphasizes a hook system for tool calls, enabling them to operate asynchronously and improving the overall computational efficiency. As this technique evolves, it promises to offer valuable solutions for managing tool dependencies and enhancing the performance of AI-driven applications.