InferenceMAX: LLM Inference Daily Benchmarks (inferencemax.semianalysis.com)

InferenceMAX is a new, continuously updated benchmarking suite that runs nightly inference tests of popular LLMs across major hardware platforms and the latest software stacks. Rather than publishing a single static score, it systematically sweeps tensor-parallel sizes and maximum concurrent requests for each model/hardware pair and produces throughput-vs-latency curves, offering a complete picture of operational trade-offs. The project uses broadly applicable serving configurations (to avoid unrealistic, highly tuned setups), and the code and results are open-sourced to encourage community contributions and reproducibility. For the AI/ML community this matters because inference performance is rapidly changing as models, libraries and runtimes evolve; static benchmarks quickly become obsolete or are gamed with niche configurations. Nightly runs plus parameter sweeps let practitioners see real-world trade-offs (e.g., higher throughput at the cost of latency under certain tensor-parallel and concurrency settings) and make informed deployment choices. By standardizing realistic software configs and making data public, InferenceMAX can improve comparability across hardware, guide cost/latency optimization, and surface performance regressions as ecosystems develop.