Vibe Code Bench (www.vals.ai)

Vibe Code Bench is a new end-to-end benchmark that evaluates large models’ ability to build complete web applications from natural-language specifications over long horizons (up to 5 hours or 1,000 turns). It runs models in isolated Docker-in-Docker developer environments with preconfigured sandbox services (Supabase, Stripe test mode, MailHog), ~30 tools (file edit, bash, SQL, browser, etc.), and automated UI testing via an autonomous browser agent. Rankings show Anthropic’s Claude Sonnet 4.5 (Thinking) and OpenAI’s GPT‑5.1 leading by a wide margin—GPT‑5.1 notable for combining high performance with low cost—but no model reliably passes every test on the first attempt and most model outputs score in the 0–12.5% range, with only a long tail of higher-performing samples from the top systems. Technically rigorous: each spec is a short natural-language app description (e.g., “Zeeter” microblog) paired with 20–60 triple-verified tests; an app’s score is the fraction of tests where ≥90% of substeps succeed. The pipeline is fully automated, reproducible, and validated against human judges (90%+ alignment), costing $10–20 per app. Error analysis highlights common failure modes—dependency installation, Docker env var/networking for backend endpoints, timeouts, premature submission, and direction-following lapses—underscoring that robust long-horizon tool use, configuration management, and test-driven iteration remain frontier capabilities for LLMs. The benchmark provides a practical framework to measure progress toward true zero-to-one AI-driven development.