BenchBench (www.strangeloopcanon.com)

A groundbreaking benchmark called BenchBench has been introduced, which evaluates the ability of AI models to create their own benchmarks and measure their performance. As AI models like GPT-5.2, Opus 4.6, and Gemini 3.5 evolve and achieve high accuracy on existing benchmarks, the challenge now lies in generating new, effective benchmarks. BenchBench aims to solve this by prompting models to propose benchmarks capable of outsmarting frontier solutions while also assessing their self-awareness and creativity in problem-solving. Significantly, GPT-5.2 emerged as the only model to create a genuinely useful benchmark, demonstrating its superiority in this new evaluation landscape. Other models, such as GPT-5.4 and Gemini 3.1, struggled to propose challenging benchmarks, often resulting in trivial or unsolvable problems. This disparity highlights a newfound distinction between models' proficiency as solvers versus creators. BenchBench not only critiques the models' capabilities in developing benchmarks but also emphasizes the importance of evaluating their creativity and self-knowledge. As a fresh evaluation framework, it promises to uncover innovative benchmarking methods to fill existing gaps in AI testing and development.