OptiLLM: Accuracy improvements on reasoning tasks with zero training (github.com)

OptiLLM is an OpenAI‑API‑compatible inference proxy that applies 20+ state‑of‑the‑art inference‑time techniques (e.g., Best‑of‑N, Chain‑of‑Thought variants, self‑consistency, MCTS, Mixture‑of‑Agents (MOA), CePO from Cerebras, MARS multi‑agent reasoning, Z3 theorem proving, PlanSearch, LongCePO, AutoThink and more) to boost reasoning accuracy without any model fine‑tuning. As a drop‑in replacement for OpenAI‑compatible endpoints (or self‑hosted via LiteLLM/Docker), it routes requests through configurable optimization pipelines (prefixing model names with a slug like moa‑model or passing optillm_approach in the request) and can produce 2–10x accuracy gains on math, coding and logical benchmarks by spending more compute at inference time. Reported highlights include +30 points on AIME 2025 (Gemini 2.5 Flash Lite), +18.6 on Math‑L5 (Llama 3.3 70B), MOA making gpt‑4o‑mini match GPT‑4 on Arena‑Hard, and +20% pass@5 on LiveCodeBench with PlanSearch. For the AI/ML community this reframes the cost/accuracy tradeoff: you can often beat “frontier” models by composing smarter inference strategies rather than training larger models, which lowers barriers to high‑quality reasoning but shifts cost, latency and energy to runtime. Practical implications include easier experimentation and production deployment (SSL, plugins for memory/privacy/code execution, multi‑provider support), but also new evaluation challenges—benchmarking must account for inference‑time ensembles and search, and teams should weigh higher inference compute and complexity against model retraining.