AI-Written CUDA Kernels Outperforms Nvidia's Best Matmul Library (www.rohan-paul.com)

Recent research has demonstrated a groundbreaking advance in AI-generated CUDA kernels, with the introduction of CUDA-L2, which produces GPU code for matrix multiplication that outperforms NVIDIA’s highly-optimized cuBLAS/cuBLASLt library by 10% to 30%. By utilizing reinforcement learning (RL) guided by large language models (LLMs), CUDA-L2 can explore a vast range of configuration spaces to generate personalized kernels that adapt to different matrix sizes and computational architectures. This capability allows CUDA-L2 to consistently yield performance improvements across various deep learning tasks, ultimately reducing the costs and time required for training machine learning models. The significance of this development lies in its potential to streamline the GPU-heavy processes prevalent in LLM training, which often rely heavily on matrix multiplication operations. With CUDA-L2’s ability to automatically generate high-performance kernels tailored to specific operational conditions, it paves the way for optimizing future GPU operations beyond just matrix multiplication—potentially enhancing performance in areas such as attention blocks or Mixture of Experts layers. By democratizing the kernel optimization process, this innovation not only challenges current conventions but also empowers developers without extensive CUDA expertise to leverage more efficient computational strategies.