Boosting multimodal inference performance by >10% with a single Python dict (modal.com)

A recent update in SGLang, version 0.5.10, has led to a significant performance enhancement in multimodal inference, achieving over a 10% improvement in both throughput and latency. This enhancement results from optimizing the management of shared GPU memory by replacing complex bookkeeping with a straightforward caching mechanism, specifically a Python dictionary. The adjustments were made following a performance analysis during the benchmarking of a multimodal model, Qwen2.5-VL-3B-Instruct, indicating that inefficient scheduling on the host side was limiting throughput below the capabilities of the GPU. This development is particularly important for the AI/ML community as it addresses inherent challenges within inference engines when handling multimodal workloads, which combine visual and textual data. The new caching technique not only streamlines the input processing but also prevents unnecessary overhead during tensor sharing across processes, enhancing overall efficiency. The results demonstrate up to a 16% increase in request throughput and a reduction in mean end-to-end latency by approximately 10%. This optimization is a crucial step towards realizing the full potential of multimodal models, facilitating more efficient applications in areas such as document parsing and multimodal coding agents.