MetalBench – Benchmark for Apple Silicon's Metal Shading Lang (github.com)

A new benchmark tool called MetalBench has been launched, designed specifically for evaluating the performance of Apple Silicon's Metal Shading language. This initiative stems from efforts to optimize inference on Apple’s M-series chips, featuring an innovative agent-based system dubbed Agent Steel. This system includes three specialized agents: a Profiler that analyzes performance data, an Optimizer that develops new kernel candidates based on this analysis, and a Verifier that tests each iteration's performance and accuracy. The benchmark supports a variety of kernel operations, allowing developers to compare against Apple’s MLX (Metal Learning Experience) performance baseline. MetalBench is significant for the AI/ML community as it offers comprehensive tools for optimizing GPU kernels, which is crucial given the growing reliance on Apple Silicon for machine learning tasks. The benchmark not only measures speed and throughput but also evaluates memory bandwidth and stability, presenting results in a user-friendly report format. Developers can now contribute to the repository with their kernels, benefiting the broader community by enhancing optimization practices for Apple's unique hardware architecture. This release emphasizes the potential for improved performance and accuracy in machine learning applications running on Apple devices.