LLM-optimizer: Benchmark and optimize LLM inference across frameworks with ease (github.com)

llm-optimizer is a new open-source Python tool designed to benchmark and optimize inference performance for any open-source large language models (LLMs) across multiple frameworks, including SGLang and vLLM. It automates finding the best configuration for your workload by testing a wide range of server- and client-side parameters—such as tensor/data parallelism, batch sizes, and concurrency—while respecting user-defined Service Level Objective (SLO) constraints like latency or throughput thresholds. This eliminates the tedious trial-and-error typically involved in tuning inference deployments. One standout feature is its theoretical performance estimation mode, which predicts latency, throughput, and concurrency limits without running full benchmarks, speeding up the exploratory phase of optimization. Results are exportable in JSON format and can be interactively explored through a Pareto frontier visualization dashboard, allowing users to intuitively analyze trade-offs between performance metrics. The tool also supports customized server commands for advanced control and integrates seamlessly with GPUs like A100, H100, and others. By enabling systematic and constraint-driven benchmarking across different LLM inference frameworks, llm-optimizer empowers AI practitioners to deploy large models more efficiently, balancing hardware usage and response times. Maintained by the BentoML team, it welcomes community contributions and promises ongoing enhancements to make large-scale LLM inference tuning more accessible and reproducible.