Reward Guided Speculative Decoding (arxiv.org)

Researchers have introduced Guided Speculative Inference (GSI), a cutting-edge algorithm designed to enhance reward-guided decoding in large language models (LLMs). GSI merges soft best-of-$n$ test-time scaling with a reward model and speculative sampling from a smaller auxiliary model to optimize the decoding process. This method effectively approximates the optimal tilted policy and expected rewards, offering a more efficient approach to generating responses from LLMs. Significantly, GSI demonstrates superior performance across various reasoning benchmarks—including MATH500 and OlimpiadBench—by achieving higher accuracy compared to traditional methods while also reducing end-to-end latency by up to 28%. By leveraging speculative samples and a reward model, the algorithm not only boosts output quality but also streamlines computational efficiency, marking a substantial advancement for the AI/ML community in developing responsive and effective LLMs. The algorithm's implementation code is publicly available, encouraging further research and application in the field.