A Unified and Diverse Benchmark for Speculative Decoding (huggingface.co)

A new benchmark called SPEED-Bench has been introduced to enhance the evaluation of Speculative Decoding (SD) algorithms, addressing previous limitations in SD assessment, which often relied on narrow datasets and conditions that do not accurately mimic real-world applications. Speculative Decoding leverages a lightweight draft model to predict multiple future tokens that are then verified by a target model, significantly improving throughput while maintaining output quality. Due to existing benchmarks focusing on limited prompt sets and short input sequences, the quality and speed of SD performance could not be measured effectively across diverse scenarios. SPEED-Bench offers a unified framework that aggregates semantic diversity into 11 categories, drawing from 18 datasets to create a robust testing environment with 880 prompts, ensuring comprehensive evaluation across various application domains. It separates the evaluation into a "Qualitative" data split, assessing speculation quality, and a "Throughput" split, which measures system performance under realistic demand metrics. By standardizing inputs and integrating with production-grade inference engines, SPEED-Bench allows for precise analysis of SD behaviors in different contexts, making it a valuable tool for researchers and developers in the AI/ML community aiming to optimize and compare SD algorithms efficiently.