Guess-Verify-Refine: Data-Aware Top-K for Sparse-Attention Decoding on Blackwell (arxiv.org)

NVIDIA has announced a significant advancement in sparse-attention decoding with the introduction of the Guess-Verify-Refine (GVR) algorithm, optimized for their Blackwell architecture. Traditional Top-K selection methods, crucial for identifying key-value entries in long-context LLMs, have often been a latency bottleneck, despite existing optimizations. GVR addresses this issue by leveraging temporal correlations across consecutive decoding steps. By utilizing the Top-K results from previous steps as a predictive signal, the algorithm efficiently narrows down candidates and verifies them without additional ballots, resulting in substantial performance improvements while ensuring exact outputs. This innovation not only achieves an impressive average speedup of 1.88x compared to existing methods but also enhances overall performance in end-to-end latency, notably improving TPOT by up to 7.52% for 100K context sizes. The GVR's design, integrated into the TensorRT-LLM stack and validated on real workloads, demonstrates the potential to enhance a wide range of sparse-attention decoders, where the Top-K process remains stable across decoding phases. The implications for the AI/ML community are profound, as this algorithm could enable more efficient and faster LLM applications across various domains.