RayTention – Self-Attention via Geometric Signal Extraction (github.com)

RayTention, a novel attention mechanism currently under U.S. patent application, offers a breakthrough for AI/ML models by addressing the significant VRAM consumption issues associated with traditional Key-Value (KV) caches. Instead of storing extensive KV records that grow with each token, RayTention compresses the entire context window into just 7 geometric signals, resulting in a fixed-size vector of 642 floats regardless of context length. This innovation allows models to operate with dramatically lower memory demands—showcasing a decrease from 4.4 GB at 1 million tokens to a mere 2.6 KB, enabling approximately 160 times more concurrent requests on the same GPU. By employing L2 distance calculations instead of the conventional dot-product method for assessing token relevance, RayTention extracts interpretable features that preserve critical information while eliminating the need for the ever-expanding KV cache. The introduction of this architecture is significant for long-context inference and edge deployments, where computational resources are limited. Furthermore, the approach enhances interpretability and simplifies transformer architectures, suggesting a promising direction for future AI frameworks, particularly in environments requiring high scalability and efficiency.