InferenceBench: A Benchmark for Open-Ended Inference Optimization by AI Agents (inferencebench.ai)

InferenceBench has been introduced as a benchmark designed to optimize open-ended inference tasks using AI agents. The benchmark evaluates the performance of various coding agents in optimizing large language model (LLM) serving workloads under a specified compute budget. Results show that AI agents consistently outperform the standard PyTorch baseline as well as most default configurations of popular inference engines, although they lag behind simple hyperparameter tuning strategies. The benchmark encompasses four scenarios — long-context prompts, long generations, concurrent traffic, and balanced serving — each assessed based on specific latency and throughput metrics. This benchmark is significant for the AI/ML community as it emphasizes the importance of not just discovering optimization techniques but also executing, validating, and preserving effective configurations in the final submissions. The benchmark encourages a structured approach to automated R&D by rewarding agents that demonstrate consistent performance improvements, thus addressing a common challenge in model deployment. Insights from the benchmark reveal that even when agents recognize potential enhancements, they often struggle to maintain optimal configurations in their final outputs, raising questions about the reliability of automated optimization in AI workloads.