The GPU never reads the same text twice (www.techradar.com)

Belgian startup Corbenic AI has launched the beta version of Galahad, a novel system designed to optimize GPU memory usage by saving a model's key-value (KV) cache to encrypted local NVMe storage. This approach allows the server to quickly reload previously processed text, significantly reducing the need for the GPU to reread documents after memory is cleared or the server restarts. Founder Sietse Schelpe emphasizes that the solution requires no specialized hardware, leveraging existing NVMe drives to enhance AI performance while promoting security by isolating customer data. Galahad aims to offer a competitive edge in a crowded market already populated by solutions like LMCache and Nvidia's Dynamo. Its notable claim of achieving a 14x speedup is primarily attributed to a component called Blaise, which intelligently retrieves only the necessary sections of text, rather than loading entire documents. While the system positions itself against retrieval-augmented generation (RAG) methodologies, it focuses on exact and secure KV persistence, potentially appealing to data centers looking to maximize their current hardware usage amidst rising memory demands. However, independent testing and benchmarks will be vital to validate its claims against established competitors.