SSV: Sparse Speculative Verification for Efficient LLM Inference (arxiv.org)

A new framework called Sparse Speculative Verification (SSV) has been introduced to enhance the efficiency of long-context large language model (LLM) inference. SSV integrates speculative decoding and dynamic sparse attention, two techniques that, when combined, have historically faced a structural mismatch. Speculative decoding benefits from cross-query commonality, while dynamic sparse attention is tailored to individual queries, limiting the efficiency of key-value (KV) cache usage. The SSV framework addresses this issue through innovative methods such as overlap-aware grouped-query execution and profile-guided orchestration, which improve cross-query reuse and reduce operational overhead. This advancement is significant for the AI/ML community as it can dramatically speed up LLM inference processes, achieving up to 3.49 times the end-to-end throughput over previous autoregressive models and delivering kernel speedups of up to 6.86 times for sparse speculative verification on NVIDIA H100 GPUs. With improved efficiency in handling queries, SSV not only enhances the performance of LLMs but also opens up possibilities for further innovation in model design and deployment, making it an important development in the quest for real-time language processing capabilities.