Show HN: Carmen – harness for reliable AI-written Metal kernels (github.com)

An innovative project called carmen has emerged that allows for the generation and testing of AI-written GPU kernels tailored for Apple Silicon. Using a large language model (LLM) to write the kernels, carmen rigorously tests each one to ensure correctness and performance enhancements, marking a significant advancement in automated kernel generation. Unlike traditional benchmarks that often overlook critical precision bugs, carmen's testing framework identified all 75 intentionally embedded bugs in kernel codes, proving far superior to existing checks. Interestingly, while the kernels showed impressive speed improvements—up to 1.82x faster than Apple's MLX on specific tasks—the overall enhancement to real model performance was minimal, primarily yielding only 2-3% faster processing times for smaller models. The significance of carmen lies in its dual approach: it prioritizes thorough testing to capture potential faults and reassesses performance gains against real-world model applications. For instance, speculative decoding emerged as a far more impactful method for speeding up larger models, achieving a 1.48x increase in speed for a 3B parameter model. This project not only highlights the potential of LLMs in optimizing GPU compute tasks but also underscores the importance of robust testing frameworks in the AI/ML community for deploying genuinely efficient AI solutions. Users can easily leverage carmen’s capabilities to measure and run models without the need for extensive setup or API keys, making high-performance model optimization accessible to a broader audience.