SGLang – High-performance serving framework for LLM's and multimodal models (github.com)

SGLang has launched an open-source inference framework designed for large language models, vision-language models, and diffusion models, catering specifically to agentic workloads and large-scale serving environments. This framework is equipped with a built-in image and video generation engine, SGLang Diffusion, which enhances its capabilities in multimedia content creation. Users can easily get started by pulling a Docker image or installing it in a Python environment, allowing for versatile deployment across various hardware platforms including NVIDIA, AMD, Google TPU, and Apple Silicon. The significance of SGLang lies in its ability to integrate a multitude of training and deployment functionalities, such as speculative decoding and reinforcement learning rollouts, in a unified framework. Additionally, its support for diverse hardware platforms facilitates broader accessibility for researchers and developers in the AI/ML community. By encouraging contributions and facilitating educational resources, SGLang aims to drive innovation and efficiency in model serving and orchestration, positioning itself as a valuable tool for AI labs, enterprises, and academic institutions.