I used autoresearch to improve my AGENTS.md, measured against real tasks (www.stet.sh)

A recent experiment involved using Codex to iteratively improve an AGENTS.md document, which serves as part of the runtime behavior for coding systems. The author implemented eight iterations, benchmarking each version against real tasks from their repository. While one version demonstrated improvements in local craft scores and managed to fix missed tasks, it ultimately regressed on a clean holdout, with increases in footprint, token usage, and declines in code-review correctness. This highlights the critical need to evaluate changes to shared documents like AGENTS.md, as outcomes can vary significantly across different tasks. The significance of this study lies in its methodological approach to understanding agent behavior through systematic testing and iteration. The author emphasizes that relying on "vibe coding" can lead to overlooked regressions, urging the AI/ML community to adopt a more rigorous framework for making updates. The investigation underscores that improvements may benefit some capabilities while harming others, reinforcing the importance of carefully measuring the implications of changes before rolling them out in shared coding environments. The findings suggest that structuring updates based on demonstrated task performance, rather than assumptions, is vital for fostering reliable AI agents.