InferenceMAX – open-source Inference Frequent Benchmarking (github.com)

InferenceMAX™ is an open-source (Apache 2.0) automated benchmarking platform that runs nightly head‑to‑head tests of the most popular open-source inference frameworks and models, publishing a live dashboard that tracks real-world inference performance as software stacks evolve. By re-benchmarking constantly rather than at a single point in time, InferenceMAX captures continuous gains from software-level advances—kernel optimizations, scheduler and distributed-inference strategies—that incrementally improve throughput, cost-efficiency and energy use between hardware step-changes. For the AI/ML community this matters because inference performance is governed by both hardware and rapidly moving software; static benchmarks go stale fast. InferenceMAX reports practical metrics such as token throughput, performance-per-dollar, and tokens-per-megawatt across vendors (AMD, NVIDIA, TPUs, Trainium) and frameworks (vLLM, SGLang, TensorRT‑LLM, etc.), with industry support for compute validation. The project emphasizes reproducible CI/CD-style benchmarking pipelines and multi-vendor transparency, helping researchers, SREs and datacenter operators make informed choices about stacks and deployments. The effort also invites collaboration (and hiring) to scale benchmarking infrastructure, making it a living reference for inference optimization progress.