TokenGate: Adapting LLMs for pre-fill with learned skimming for ~11x speedup (github.com)

TokenGate has been introduced as a novel component for adapting large language models (LLMs) to enable rapid pre-fill operations through learned skimming of input tokens. By selectively reading only the most relevant tokens, TokenGate achieves extraordinary performance improvements, boasting up to 11.3 times faster pre-fill times and approximately 3.4 times lower memory usage compared to traditional methods. This development is significant for the AI/ML community as it enhances model efficiency without compromising accuracy, offering a unique solution for handling long-context inputs. Technically, TokenGate acts as a lightweight, drop-in extension for existing models, allowing for conditional computing reminiscent of Mixture of Experts (MoE) layers. It features a mechanism that enables end-to-end training with standard prediction tasks while utilizing a non-differentiable top-K token selection process. This method ensures that tokens deemed important are processed through the subsequent layers, drastically reducing both computation time and memory footprint. The framework simplifies integration into pre-existing architectures, making it accessible for developers familiar with popular libraries like Hugging Face Transformers. Overall, TokenGate represents a crucial advancement in optimizing LLMs, potentially setting a new standard for efficient context processing.