OpenAI's AGI number came from a harness, not the model (thenextweb.com)

OpenAI recently claimed to have entered the AGI era with its latest model, GPT-6 Astra, achieving a staggering 99.9% score on the ARC's benchmark. However, further scrutiny revealed that this high score originated from a specialized software "harness," the Provider Adapter, designed to enhance the model's performance by managing its memory and context more effectively. When the same model was tested under standard conditions, it scored only 62.7%. This discrepancy highlights the significant impact that the operational context has on AI performance metrics, raising questions about the validity of comparing different results derived from varying setups. The implications for the AI/ML community are profound. As organizations continue to develop and market advanced models, the focus must shift towards understanding the systems that surround these models—how they are structured, configured, and evaluated. This incident underscores a critical gap between model capabilities and independent verification, emphasizing the need for transparency in reporting metrics. ARC Prize co-founder Mike Knoop explicitly stated that the organization does not endorse OpenAI's AGI assertion, suggesting that much remains to be understood about the capabilities of models like Astra. As the landscape of AI continues to evolve, the creation and sale of harnesses as products could redefine performance assessments, complicating both public perception and academic discourse surrounding AI advancements.