Rethinking LLM Serving with System One Models (supercomputing-system-ai-lab.github.io)

A new approach to optimizing large language model (LLM) serving is introduced with the concept of "System One models," which aim to streamline decision-making processes in LLM services. Traditionally, serving requests involves a series of rigid, rule-based decisions that may be fast but do not account for the nuances of different requests. In contrast, System One models provide a cost-efficient and quick method to evaluate decisions by delivering probabilistic responses to multiple questions simultaneously, thus enhancing efficiency without the overhead of generating lengthy text outputs. This advancement is significant for the AI/ML community as it integrates fast, predictive capabilities into LLM serving systems. The System One model, exemplified by the Jev 1.13, boasts impressive response times of around 92 milliseconds and can handle multiple queries in a single pass. By refining the decision matrix within the serving stack, such as prioritizing shorter requests based on predicted output length, these models promise to increase throughput by 15-21% while maintaining low latency. Furthermore, systematic benchmarking through JevServe-Bench allows developers to measure and compare the decision-making efficacy across various models, highlighting important areas for future study and optimization.