Client-side GPU load balancing with Redis and Lua (galileo.ai)

A team at Galileo has enhanced GPU utilization for AI workloads by implementing a client-side load balancer leveraging Redis and Lua, achieving a remarkable 40% increase in average GPU utilization and a 70% reduction in tail latency. Traditional Kubernetes load balancers operate on round-robin algorithms, which are ineffective for the highly variable latency associated with GPU inference tasks, leading to inefficient resource use. The new load-aware approach ensures that each inference request is directed to the least busy GPU, optimizing workload distribution and reducing latency. By adopting a client-side model, the implementation not only minimizes network hops—thereby reducing latency—but also localizes failures to individual clients, enhancing system resilience. The load balancer estimates the GPU workload based on the byte size of the inference request, facilitating a straightforward and efficient routing process. Utilizing Redis for shared state management and load tracking, the system employs atomic updates through Lua scripts to maintain accuracy and consistency, significantly improving operational efficiency in real-time AI evaluation scenarios. This innovation highlights an essential shift in how resource-heavy AI models can be managed effectively, addressing challenges often faced in high-demand environments.