Three types of LLM workloads and how to serve them (modal.com)

A recent article discusses the classification of large language model (LLM) workloads into three categories: offline, online, and semi-online, presenting tailored architectural recommendations for each to optimize performance and cost. This classification is crucial as organizations increasingly shift from proprietary model APIs toward open-source solutions. Innovations from companies like DeepSeek and Alibaba have contributed to a more nuanced approach to handling LLM workloads, challenging the flat pricing models typically offered by API services and emphasizing the need for engineers to understand specific workload requirements better. For offline workloads, characterized by batch processing and high throughput needs, the article recommends leveraging the vLLM engine with asynchronous remote procedure calls (RPC) to maximize cost efficiency. Conversely, online workloads, which require low-latency interactions, benefit from using SGLang with advanced architectural strategies that minimize host and communication overhead. Semi-online workloads, which fall somewhere in between, demand flexible infrastructure that can quickly scale up or down based on real-time needs. By addressing the unique challenges each workload type presents, the article equips AI engineers with the knowledge to build more efficient LLM applications, reflective of shifting trends in the industry.