VLLM Predicted Outputs (cascadetech.ai)

Cascade Technologies announced a new implementation of Predicted Outputs for the open-source VLLM inference stack, letting clients supply an expected output (for example, the original source file when asking for an edit) so the model can skip regenerating unchanged tokens. When the supplied prediction matches the model’s true output, those tokens are processed in parallel like input tokens instead of the usual slower one-by-one output pass — producing speedups that scale nearly linearly with prediction accuracy (Cascade claims ~50% prediction ≈ half the time, 100% ≈ near-instant). They provide a demo and compare results on a ~800-token Python game (verbatim prediction acceptance ~93–97%; editing to “multiplayer” ~26–40%). Technically, Cascade treats the static user prediction as speculative proposals rather than running a separate draft model: it proposes chunks from the prediction, accepts only exact matches (so output correctness is unchanged), and when divergence occurs it realigns using Myers diff and continues generation. The prediction work runs on CPU (no extra GPU), can hide alignment cost with one-frame delay, and plugs into standard OpenAI-style APIs via VLLM’s speculative decoding hooks. This approach is especially useful for code-modifying agents, structured-output workflows, document edits, and agent state updates where much of the output is unchanged, enabling large practical speedups without sacrificing accuracy.