Shader Benchmark for LLMs (nbardy.github.io)

A recent study evaluated three advanced coding agents—Claude Opus 4.7, Gemini 3.1-pro-preview, and Codex GPT-5.5 high—on their ability to generate WebGPU Shading Language (WGSL) shaders from text prompts targeting 130 mathematical visualization problems. The models were scored on a scale of 0 to 100 across five categories by one or more judging large language models (LLMs), with scores revealing that performance varied widely. For instance, Claude Opus scored an average of just 12%, with significant failure to render images, while Gemini achieved the highest score with approximately 29% when excluding failures. This benchmarking is significant for the AI/ML community, as it highlights the current capabilities and limitations of LLMs in generating functional code for complex mathematical visualizations. The study notes high failure rates in shader compilation, which impacts the overall effectiveness of these models in practical applications. These findings provide a valuable framework for further research into improving model architecture and training strategies, emphasizing the need for more robust solutions to enable successful code generation in real-world scenarios.