KernelBench: Can LLMs Write GPU Kernels? – Benchmark and Toolkit, Torch –> CUDA (github.com)

KernelBench has launched as a new benchmark toolkit aimed at assessing the capability of large language models (LLMs) in generating efficient GPU kernels. By focusing on generating correct and performant CUDA kernels specifically for PyTorch programs, KernelBench offers a structured evaluation environment divided into four distinct categories, ranging from single-kernel operators to full model architectures. The toolkit not only facilitates the evaluation of generated kernels against reference PyTorch operators for correctness but also measures their performance, introducing a novel metric, fast_p, which gauges the fraction of tasks achieving both correctness and speedup. This initiative is significant for the AI/ML community as it opens avenues for optimizing model deployment on GPUs, potentially enhancing performance and efficiency in various applications. Its focus on generative capabilities of LLMs in the context of hardware optimization could drive advancements in automated programming and model optimization pipelines. Additionally, the expanding support for multiple DSLs and AMD GPUs underlines its versatility, attracting both researchers and developers interested in utilizing AI for GPU kernel generation and optimization. As KernelBench continues to evolve, it promises to be a valuable resource for fostering innovation in hardware-software co-optimization.