Coda-GQA-L Bounded Memory Differential Attention with Value-Routed Landmark Bank (huggingface.co)

The introduction of CoDA-GQA-L represents a significant advancement in transformer architecture, specifically addressing the memory limitations associated with longer context sequences. By replacing the traditional O(L) key-value (KV) cache with a fixed-size, three-segment memory buffer—comprising a recent window, an exact landmark bank, and a summary bank—this new mechanism significantly compresses memory usage from 160 GB to just 136 MB for a 70B model. This breakthrough allows the model to efficiently manage longer contexts without exceeding memory limits, crucial for deploying large models on commodity hardware. Key technical innovations include the use of differential attention via orthogonal query rotation, which maintains efficient attention computations while sidestepping traditional position-dependent constraints. The model utilizes learned gating and semantic routing for memory management, ensuring that every slot in the cache is effectively utilized. By evaluating tokens based on their semantic similarity rather than positional encoding, CoDA-GQA-L optimizes the retention of important tokens and reduces the computational overhead typically associated with contextual memory. This architecture positions CoDA-GQA-L as a transformative solution to long-sequence processing in AI, promising to enhance the scalability and applicability of large-scale language models.