Do Not Optimize Away (matklad.github.io)

A new discussion in the AI/ML community highlights the challenges of benchmarking code due to compiler optimizations, particularly with LLVM. When complex calculations are performed, LLVM can replace them with simplified forms to enhance performance, which is beneficial in deployment but problematic for accurate benchmarking. The article emphasizes that this optimization can lead to misleading benchmark results if the compiler eliminates unnecessary operations entirely. To address this issue, the author suggests a more straightforward approach that avoids the pitfalls of using explicit optimization hints found in languages like Rust or Zig. By ensuring that input parameters are dynamic during benchmarking and computing a "hash" of the results, the compiler is less likely to optimize away crucial parts of the code. This method not only prevents the erasure of meaningful computations but also provides a safeguard: discrepancies in the hash value serve as an immediate warning that optimizations may have altered the correctness of the output. This straightforward technique provides a practical solution for developers seeking reliable benchmarks.