Alibaba Cloud says it cut Nvidia AI GPU use by 82% with new pooling system (www.scmp.com)

Alibaba Cloud unveiled Aegaeon, a GPU pooling system that cut the number of Nvidia H20 GPUs required to serve dozens of LLMs (up to 72 billion parameters) by 82% in a beta run—dropping from 1,192 GPUs to 213—according to a research paper presented at SOSP. The system was tested in Alibaba Cloud’s model marketplace for more than three months and developed jointly with Peking University (paper co-authored by Alibaba CTO Zhou Jingren). The researchers highlight extreme inefficiency in current inference deployments: 17.7% of GPUs were servicing just 1.35% of requests, driven by a heavy skew where a few popular models handle the bulk of traffic while many others are rarely invoked. Technically, Aegaeon pools GPU resources so a single GPU can serve multiple concurrent model workloads, improving utilization and lowering provisioning overhead for sporadically used models. For the AI/ML community this signals a practical path to reshape inference economics—reducing hardware costs, enabling cloud providers to host more models concurrently, and changing capacity planning and pricing for model-serving platforms. The work also raises follow-on questions about latency, scheduling, memory management and isolation when multiplexing large models on shared accelerators, which will be crucial for broader adoption.