Tau² Benchmark: How a Prompt Rewrite Boosted GPT-5-Mini by 22% (quesma.com)

Using the new Tau² benchmark (telecom_small, 20 scenarios) the team found a simple prompt rewrite raised GPT-5-mini’s one-shot success rate from 55% to 67.5% (a 22.7% relative boost) and improved pass^2 (reliability with retries) from 40% to 50% (+25%). GPT-5-mini is roughly twice as fast and ~5× cheaper than flagship GPT-5, and initially delivered 85–95% of GPT-5’s capabilities; after optimization it overtook an o3 baseline (58%) and moved closer to GPT-5 (~97%). The process also cut the number of tasks that failed in every trial from six to three, showing the change unlocked capabilities previously deemed unsolvable for the smaller model. Technically, the team used Claude to rewrite domain agent policies (main_policy.md, tech_support_manual.md) into AI-optimized, checklist-style prompts: clear decision trees, explicit prereqs, deterministic yes/no branches, precise tool-call specs, error-handling and verification steps, and reduced cognitive load. The takeaway for ML practitioners is practical: structured, imperative agent prompts — and using a stronger model to automatically refactor policies — can materially close performance gaps for lightweight LLMs, improving accuracy, reliability, and cost-efficiency in agentic workflows.