Eagle-4: Speculative decoding drafter for vLLM and SGLang (github.com)

EAGLE-4, a novel extrapolation algorithm specifically designed for enhancing the efficiency of large language model (LLM) inference, has been introduced. This new framework utilizes a flexible drafting system that dynamically adapts to various batch sizes and hardware configurations, considerably improving inference speed. Notably, when run on four NVIDIA B300 GPUs, EAGLE-4 achieves an impressive 3.40× speedup for processing single requests compared to SGLang's traditional methods without speculation. The technology is integrated within the eagle-harness package, facilitating compatibility with platforms like SGLang, vLLM, and TokenSpeed, offering developers streamlined deployment options. The significance of EAGLE-4 for the AI/ML community lies in its potential to dramatically enhance the performance of LLM applications, reducing latency and resource consumption, which is crucial as models continue to scale. By keeping the output distribution of the target model intact while optimizing the draft tree, EAGLE-4 not only expedites processing times but also maintains accuracy. The technical implications are substantial, enabling users to scale their machine learning workloads more effectively across different setups, from cloud-based infrastructures to dedicated hardware, thereby driving innovation and productivity in various AI applications.