TokiRecur – llama.cpp LLM launcher with KV-cache recycling (github.com)

TokiRecur has launched a cutting-edge tool that optimizes local inference for large language models (LLMs) through a novel approach to reusing key-value (KV) caches. Instead of rebuilding the KV cache from scratch for every new prompt—often containing significant overlap with previous prompts—TokiRecur intelligently identifies and reuses exact cached prefixes. This mechanism significantly boosts throughput: benchmark tests demonstrate improvements in time-to-first-token (TTFT) by 62-96% and prefill throughput enhancements by as much as 1280%, particularly benefitting larger mixture-of-experts (MoE) models. This innovation is critical for the AI/ML community as it promises to enhance the efficiency and speed of LLM operations, thereby facilitating more usable and responsive AI applications. The technical implementation operates within various platforms using multiple backends like CUDA, ROCm, and more, while ensuring lossless output integrity. Notably, the cache is ephemeral and exists in memory, posing certain limitations regarding model restarts and configurations, as the KV cache can only be restored within a single session. This focus on precise and efficient utilization of resources marks a substantial leap in local LLM inference technologies, evidencing TokiRecur's potential to influence future AI system designs significantly.