SCache and MOS – 32.9k O(1) KV cache (22.7 MiB) and 72B INT4/ASVD engine (github.com)

Researchers have announced a groundbreaking advancement in AI memory management with the introduction of Mixture of Scars (MoS) and SCache, which tackle significant hardware challenges associated with large autoregressive transformer models. Specifically, these innovations address the growing size of parameter storage and inefficient key-value (KV) cache mechanisms, enabling systems to function more efficiently with Transformer models up to 72.71 billion parameters while reducing memory requirements from 135.43 GiB to just 38.12 GiB. This approach employs a novel single-directional residual formulation, effectively neutralizing model outlier responses and enhancing overall memory efficiency. This advancement is crucial for the AI/ML community as it paves the way for deploying extremely large models on consumer-grade hardware, such as the Apple Silicon M2, which can now handle a 72B model without overloading its memory resources. The technical framework relies on pairs of INT4 base models and Activation-Aware SVD (ASVD) techniques. Additionally, SCache offers a lightweight, dynamic cache architecture that maintains constant memory growth, improving throughput and reducing the load on physical memory. These innovations not only push the boundaries of what is achievable in AI but also ensure scalability and accessibility for future AI research and applications.