LLM That Beat Frontier Models at Chess (www.aidancooper.co.uk)

In an intriguing showdown in the AI chess realm, OpenAI's GPT-3.5 Turbo Instruct has emerged victorious against advanced frontier models like GPT-5.6 Sol and Claude Opus 5, clinching ten consecutive wins. As OpenAI plans to discontinue access to GPT-3.5, the impressive undefeated record raises questions about the capabilities of newer models and the conditions under which they operate. GPT-3.5, while instruction-tuned, showcased exceptional performance by processing complete game records as PGN, enabling it to effectively predict the next move based on a realistic context. In contrast, the frontier models struggled, making illegal moves due to their inability to track the changing game state effectively. The outcomes highlight significant implications for the AI/ML community regarding model training and performance in specific formats. The frontier models' challenges may suggest that despite their advanced architectures, without the right input format, they fail to leverage their underlying chess knowledge. This scenario raises critical discussions on how conversational training affects a model's ability to engage in structured tasks like chess and whether these newer models can be fine-tuned for better performance in competitive environments. The performance dependency on input formatting underscores the complexity of teaching AI nuanced real-world skills that extend beyond pure computation.