AoE 2 Build Order as an Eval for LLM's (wraitii.github.io)

A new benchmark test utilizing the build order for Age of Empires 2 (AoE2) has been introduced to evaluate large language models (LLMs) in coding optimization tasks. This test challenges LLMs to follow strict programming syntax and strategically coordinate multiple objectives, such as developing efficient build orders while adhering to a complex JSON format. The inclusion of challenging variables like context rot and out-of-distribution coding makes this evaluation distinct, pushing models to demonstrate their agentic coding abilities beyond conventional tasks. Despite the compelling nature of this test, results across multiple models, including Gemini 3.1 Pro, have been underwhelming; none achieved performance levels deemed satisfactory. While all tested models were able to produce a domain-specific language (DSL) script, they fell short of expectations for tasks requiring strategic planning and in-depth understanding of AoE2 mechanics. The assessment highlights a significant skill gap among LLMs, illustrating that while they excel in general coding, proficiency in specialized coding contexts remains a challenge. This benchmark not only serves to assess current capabilities but also opens avenues for improving LLM performance in complex coding scenarios.