Speeding up GPU kernels by 38% with a multi-agent system (cursor.com)

A cutting-edge multi-agent system has demonstrated a remarkable 38% average speedup in optimizing CUDA kernels, which are vital for AI training and inference on NVIDIA GPUs. Over three weeks, the system autonomously tackled 235 optimization problems across various real-world models, achieving results traditionally reserved for expert engineers. By exploring a broader solution space and employing innovative strategies, it outperformed human-optimized baselines on 149 problems and delivered over 2x speedup in 19 cases. This could significantly enhance GPU utilization, reduce energy consumption, and facilitate larger AI models, making this development highly significant for the AI/ML community. The multi-agent framework autonomously generated efficient solutions in two programming languages, CUDA C and CuTe DSL, optimizing kernel performance at both low and high abstraction levels. For instance, it optimized a grouped-query attention kernel to improve LLM performance and tackled complex matrix multiplication challenges, inching closer to professional-grade benchmarks. The study not only validates the potential of multi-agent systems in addressing complex software optimization tasks but also suggests exciting implications for future AI tools, indicating that enhanced multi-agent solutions could soon supersede human expertise in this domain.