OK-DMD – Koopman Dynamic Mode Decomposition for KV-Cache Eviction (github.com)

OK-DMD is a groundbreaking framework for optimizing KV-cache compression and eviction in Long-Context Large Language Models (LLMs), designed to enhance memory retention during lengthy text generations. Unlike traditional eviction strategies that treat attention keys as static entities, which often leads to significant context loss—particularly during domain shifts—OK-DMD models the relationship between attention keys as a dynamic flow, akin to tracking a river's current. By using Koopman Operator Theory, it maintains awareness of the stable semantic "attractors" in the text, ensuring that vital information remains accessible even as transient data is evicted. This innovative approach addresses the shortcomings of existing methods like H2O, which can discard essential context when unexpected topics arise, causing hallucinations in generated output. In rigorous testing, OK-DMD demonstrated superior performance under aggressive compression requirements, consistently preserving attention coherence and allowing LLMs to adapt to shifts in conversation without losing previous context. Key technical components include a robust matrix update mechanism and strategies for managing memory trade-offs, making it not only efficient but also a vital tool in improving LLM deployments across diverse applications.