New Research Reassesses the Value of Agents.md Files for AI Coding (www.infoq.com)

A new study from ETH Zurich challenges the conventional wisdom surrounding the use of AGENTS.md files in AI coding tasks, suggesting that these context files may actually hinder performance rather than enhance it. The researchers found that LLM-generated context files decreased task success rates by 3% and increased inference costs by over 20% due to agents taking more unnecessary steps. In contrast, while human-written files provided a slight 4% improvement in success rates, they also resulted in a significant increase in the number of steps required. The team developed AGENTbench, a novel dataset of 138 real-world Python tasks, to empirically assess the effectiveness of these files in precise coding scenarios, revealing a disconnect between current developer recommendations and the study’s findings. The implications of this research are significant for the AI/ML community, prompting a reconsideration of how context is provided to coding agents. It highlights the need for developers to focus on crafting high-quality, task-relevant instructions rather than relying on auto-generated, potentially detrimental content. The study advocates for future work aimed at creating effective methods for generating useful guidance, underlining that thoughtful documentation might be beneficial, especially in more complex, larger codebases. Developers engaged with the paper noted that clear, well-structured AGENTS.md files could serve as crucial references that improve both agent efficiency and onboarding processes for human collaborators.