We gave an AI agent eyes. It didn't even use them (www.agentvoyagerproject.com)

A recent experiment tested the capabilities of the AI model Claude Haiku 4.5 in a challenging task of extracting and reconstructing a complex PDF table using a robust agent framework called Goose. Two configurations were evaluated: one with the ability to process images (pdf-vision) and another relying solely on text (pdf_tool). Surprisingly, despite having "eyes," the vision-enabled configuration could not leverage its visual capabilities effectively, scoring only 53%. In contrast, the text-based tool succeeded in reconstructing the table perfectly, demonstrating that the strength of the agent harness played a critical role in the outcome. This experiment is significant for the AI/ML community as it highlights the importance of the agent's tools and configurations over raw model capabilities. It emphasizes that a cheaper AI model can achieve impressive results when equipped with a robust framework, even when the model itself does not utilize all its available features. The findings advocate for understanding agent trajectories, promoting transparency in AI decision-making, and challenging assumptions about the necessity of advanced tools for complex tasks. As the research team intends to continue exploring these dynamics, it shows promise for reshaping how AI agents are designed and applied in practical scenarios.