PTXBench: Benchmarking and Adapting LLMs for GPU Kernel Optimization (arxiv.org)

Researchers have introduced PTXBench, a benchmarking tool designed to evaluate and optimize large language models (LLMs) for GPU kernel performance using architecture-specific Parallel Thread Execution (PTX). This tool assesses both the functional correctness of target instructions and their execution speed against leading libraries, focusing on GEMM and attention tasks on advanced GPUs like the H100 and B200. PTXBench revealed performance disparities, particularly in complex tasks, indicating that simply executing target instructions does not guarantee competitive performance. The significance of PTXBench lies in its role as an auditable platform for enhancing LLMs' efficiency in leveraging evolving GPU technologies, addressing a crucial gap in the current AI/ML landscape. In adapting the Qwen3.6-27B model, the researchers incorporated supervised fine-tuning, finding that repair-conditioned training yielded mixed improvements across tasks, influenced by data quality and coverage. This highlights the ongoing challenges in generalizing LLMs across different workloads, underscoring the need for continuous refinement and evaluation tools like PTXBench to drive advancements in model performance on cutting-edge hardware.