AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens (github.com)

Researchers have introduced AbridgeKV, a novel approach for compressing model outputs during long reasoning tasks by utilizing learned pause tokens in the key-value (KV) cache. This innovative technique allows the model to compress its output after generating every 512 tokens by replacing sections of the output with pause tokens, which do not produce additional outputs but serve as an efficient encoding mechanism. By freeing up KV entries when these pause tokens are employed, the model significantly reduces the cache size needed per sequence—resulting in performance improvements, including a 3.2x speed increase on saturated H100 GPUs compared to traditional full KV methods. The significance of AbridgeKV lies in its ability to manage memory and computational resources more effectively within transformer models, which is crucial as AI systems tackle increasingly complex tasks. The design choice to allocate cache budgets per layer based on their utility and the option to compress larger token blocks enhances both efficiency and flexibility. With promising results demonstrated on a suite of problem-solving tasks, AbridgeKV represents an important step forward in enhancing the scalability and speed of AI reasoning processes, potentially influencing future model architectures and applications in various domains.