Benchmark that evaluates LLMs using 759 NYT Connections puzzles (github.com)

The recently updated NYT Connections LLM Benchmark now evaluates large language models using 759 puzzles, adding complexity through up to four extra trick words per puzzle. This new version addresses the nearing saturation of the original benchmark, allowing for more challenging assessments and facilitating better differentiation among models' reasoning capabilities. Notably, models like Gemini 3 Pro Preview lead with an impressive score of 96.8, marking a significant advancement for AI systems in natural language understanding and reasoning tasks. The benchmark is essential for the AI/ML community as it enhances our understanding of how advanced LLMs perform in complex reasoning scenarios, crucial for applications across various sectors. The benchmark results show that top models, particularly from OpenAI, consistently outperform the average human player in solving these puzzles, indicating a shift towards superhuman performance in specific cognitive tasks. Additionally, systematic testing against human performance metrics may redefine how we gauge AI intelligence, with implications for future LLM designs and their adoption in real-world applications.