Speculative cascades – A hybrid approach for smarter, faster LLM inference (research.google)

Google Research has introduced "speculative cascades," a novel hybrid technique that enhances large language model (LLM) inference by combining the efficiency strategies of speculative decoding and model cascades. Traditional cascades use smaller models to handle simpler queries before escalating to a larger model only when necessary, optimizing cost but facing sequential delays. Speculative decoding accelerates generation by allowing a smaller "drafter" model to propose token sequences that are simultaneously verified by the larger "expert" model, speeding up output without altering final results but sometimes increasing memory and computation demands. Speculative cascades overcome the limitations of both by using a flexible deferral rule that dynamically decides, on a token-by-token basis, whether to accept the smaller model’s draft or to defer to the larger model’s output. This nuanced approach prevents the sequential bottleneck seen in cascades while avoiding the rigid token-matching rejections typical of speculative decoding. Tested across diverse language tasks with Gemma and T5 models, speculative cascades consistently achieved better quality-speed-cost trade-offs, producing faster responses with comparable or improved accuracy. This innovation offers developers finer-grained control over inference efficiency, enabling smarter, faster LLM-powered applications. By harmonizing cascades’ cost-awareness with speculative decoding’s parallel verification, speculative cascades represent a significant step toward scalable, practical deployment of large models that meet both performance and budget constraints.