A 32K-window proxy that beat full context on long agent histories (github.com)

A new transparent HTTP proxy, dubbed "mnemonic-proxy," has been introduced to enhance coding agents by enabling them to manage extensive conversation histories without losing context. This proxy operates between a coding agent and various AI models, like llama-server and Strata, allowing the agent to send the complete and unmodified history while the proxy decides what the model actually reads. This setup retains every message verbatim on disk, enabling the model to retrieve past exchanges efficiently. In benchmarks, mnemonic-proxy demonstrated a significant performance improvement, achieving a remarkable accuracy of 86 out of 100 in a 32K token window, compared to only 47 with a typical LLM summary. The significance of mnemonic-proxy lies in its ability to handle longer interactions seamlessly, crucial for applications requiring sustained dialogue and context retention. It allows for real-time recall and archiving of conversations, drastically reducing the load time for resumed sessions—from nearly 50 seconds to just 1.4 seconds. This architectural advance not only enhances efficiency but also preserves the integrity of the original input, making it particularly appealing for developers and researchers in the AI/ML community. With its utility across various engines and compatibility with Python 3.10 or higher, mnemonic-proxy promises to revolutionize the way coding agents handle lengthy interactions, setting a new standard for context management in long-term AI applications.