Humans Still Beat AI in the Long Horizon (joyemang33.github.io)

Recent research highlights a persistent gap in long-horizon performance between humans and AI agents in coding contests. In a significant evaluation, agents like Claude Opus 4.6 and GPT-5.5 were tested against top human competitors in a two-week coding challenge. While agents initially improved rapidly during the first 24 hours, their performance plateaued, contrasting sharply with the continuous improvement shown by humans over the entire contest duration. This outcome emphasizes that current AI still struggles with long-term test-time adaptation, suggesting its underlying strategies primarily revolve around repeated sampling rather than true adaptive learning. The researchers introduced a reference model based on repeated sampling to assess agent performance, revealing that while agents' Elo ratings grow linearly with increased test-time compute, they fail to match the superlinear improvement demonstrated by human contestants. This study calls for further exploration of open-ended, long-horizon tasks to better understand the shortcomings of AI agents and to establish clearer benchmarks for their development. Ultimately, the findings underline the necessity for AI advancements to incorporate and emulate human-like adaptability in extended problem-solving scenarios.