How we test LLM serving changes for coding agents, two hours of GPUs per change (www.upbound.io)

Dennis Ramdass, Principal AI Engineer, shared insights on testing changes to Large Language Model (LLM) serving for coding agents using their internal platform, Modelplane. The team is currently experimenting with GLM-5.3 across 2 nodes equipped with 16 RTX PRO 6000 GPUs. Efforts aim to transition from a pilot group of 10 engineers to a larger team of 15, revealing that the initial setup couldn't support the larger group effectively. Testing various configurations unveiled surprising demands from light users—whose subagents can maintain numerous requests simultaneously—requiring a reevaluation of GPU allocation and context sizes. The tests revealed that while each session typically required around 2 hours on the GPU, optimizing performance would entail examining various factors including recipe layout, context limits, and prefix cache management. This exploration is significant for the AI/ML community as it provides a framework for understanding the nuanced interactions between user demand, model performance, and resource allocation in LLM systems. By measuring real user interactions alongside controlled tests, the team developed specific metrics for evaluating system capacity, such as time to first token (TTFT) and concurrent session counts. This methodological approach not only aids their internal scaling efforts but also sets a precedent for other organizations working to optimize their LLM serving infrastructure, particularly as user loads increase and conversation complexities grow.