RIS-Kernel: Running 64k context LLMs on CPU via sparse attention (github.com)

The RIS-Kernel is an innovative sparse attention inference engine designed to facilitate the execution of large language models (LLMs) with context windows exceeding 64,000 tokens on standard CPU hardware. Traditionally, self-attention mechanisms in LLMs have a quadratic scaling complexity, necessitating expensive GPU resources for large contexts. By employing Reduced Interaction Sampling (RIS), this model-agnostic approach reduces this complexity to logarithmic scale, allowing effective inference on commodity CPUs, which is significant for researchers and developers lacking access to high-end GPUs. The efficacy of RIS has been validated using the Qwen2-1.5B-Instruct model, demonstrating impressive accuracy metrics even under substantial memory constraints—achieving notable retrieval gains and outperforming native dense attention at various densities. This advancement not only makes long-context analysis feasible on common academic machines but also promotes inclusivity in AI research by reducing hardware limitations. The RIS-Kernel's architecture emphasizes stability and retrieval coherence, with practical implementations available for reproducibility in research, marking a critical step forward in scalable LLM applications.