Grinder12: 0.96-Bit Lossless Streaming KV-Cache (16.55x VRAM Savings (github.com)

Grinder12, an innovative local inference-engine research project by independent systems engineer Michael Stochl, has achieved a groundbreaking 0.96-bit lossless streaming Key-Value (KV) cache. This technology, primarily targeting transformer models within llama.cpp runtimes, boasts a remarkable 16.55x compression rate compared to traditional FP16 KV caches. In practical terms, this means that the persistent KV footprint can be reduced from approximately 11.47 GB to around 693 MB at a context of 200,000 without compromising performance, as demonstrated through rigorous testing. The significance of Grinder12 for the AI/ML community lies in its potential to dramatically enhance memory efficiency for large-scale machine learning models, thus enabling more scalable deployment on resource-constrained systems. The experimental results indicate that the streaming path maintains perfect cosine correlation and negligible divergence in performance metrics, validated by passing all 125 allotted tests. Furthermore, the technology’s controlled C++ implementation and CUDA allocation verification suggest robust foundational capabilities, paving the way for potential partnerships or further development. As Stochl seeks collaborators to independently verify these findings, the project may set a new standard for KV-cache compression in AI applications.