Why Do Astra and Opus 5.5 Fail Simple 2D Mazes? (www.fig.inc)

Recent evaluations of GPT-6 Astra and Claude Opus 5.5 on 50 interactive 2D mazes reveal significant insights into their problem-solving capabilities, particularly in comparison to their predecessors. Astra successfully solved 11 mazes while Opus 5.5 solved 7, outperforming Claude Opus 4.8's previous best of 4. Notably, both models struggled with mazes featuring switch-gate mechanisms, leading to a stall kill on all attempted episodes. This analysis highlights a shift in how task horizon impacts model performance; unlike older models, progress in Astra and Opus 5.5 is minimally affected by maze length, marking an evolution in frontier AI capabilities. The evaluations also uncovered distinct failure patterns between the two models. Astra tends to traverse already explored areas, while Opus 5.5 repeatedly attempts ineffective actions due to its inability to properly engage with the environment. Interestingly, simplified prompts improved their performance in select scenarios, suggesting that strategic instructions can enhance reasoning. These findings underscore not only the advancements in AI capabilities but also the importance of understanding the underlying mechanisms of model failures, paving the way for future innovations in AI/ML design and training.