Frontier LLMs drop from 83% to 43% once reasoning has to chain across domains (arxiv.org)

Researchers have introduced Relay-Bench, a comprehensive benchmark designed to evaluate the reasoning capabilities of large language models (LLMs) when faced with multi-domain challenges. The testing revealed that even the top-performing LLM, GPT-5.5 (xHigh), achieved only a 43.3% success rate when required to integrate reasoning across different domains. The benchmark comprises complex tasks that combine single-domain subproblems, emphasizing the model's ability to process and reason through diverse contexts and complexities, including visual reasoning, coding, mathematics, and more, all within a text-only format. This evaluation is significant for the AI/ML community as it highlights the limitations of current LLMs when tasked with intricate reasoning across multiple domains, despite their impressive performance in isolated problems. The findings demonstrate the challenges associated with context comprehension and composite problem-solving, underscoring the need for advancements in model architectures and training methodologies. By allowing models to access various tools like code execution and web searches, Relay-Bench invites further exploration into how LLMs can be optimized for complex, real-world tasks, driving the evolution of AI capabilities.