Accelerating LLM Inference with Lossless Speculative Decoding Algorithms (2025) (arxiv.org)

A recent paper introduces innovative lossless speculative decoding (SD) algorithms that enhance the inference speed of large language models (LLMs) without the prerequisite of shared vocabularies between drafter and target models. Traditional SD methods have limited practical application by requiring both models to operate with identical vocabularies, a constraint that necessitates extensive retraining. The new approaches break this barrier, allowing any pre-existing model to serve as a drafter without modifications, while maintaining the integrity of the output distribution. This advancement is particularly significant for the AI/ML community as it promises speed improvements of up to 2.8 times compared to standard autoregressive decoding across various tasks, such as summarization and long-context processing. The ability to utilize off-the-shelf models in diverse applications not only streamlines workflows but also broadens the practical implementation of speculative decoding, making it more accessible to researchers and developers. By facilitating faster inference without additional training costs, these algorithms could accelerate the development of generative AI applications and enhance overall model efficiency.