We Ran 250 AI Agent Evals to Find Out If Skills Beat Docs (www.wix.engineering)

Wix conducted a comprehensive study involving 250 evaluations to assess whether AI agents perform better using purpose-built "skills" or traditional documentation. The research revealed that optimized documentation often surpassed skills in task success rates, highlighting the critical need for well-structured, up-to-date docs tailored for AI consumption. For example, improvements in documentation led to an increase in task completion rates from 67% to 87%, while skills faltered due to minor errors and inconsistencies, which eroded their expected advantages. Significantly, the study sheds light on the evolving role of documentation as AI agents replace human developers as the primary audience for tech writing. It establishes a valuable framework where optimized documentation serves as a foundational resource, while skills act as a supplementary layer for common tasks. This nuanced understanding emphasizes that while skills can provide efficiency in specific scenarios, they have limitations that must be managed with regular evaluations and updates to maintain their relevance and utility. Ultimately, Wix's findings encourage tech teams to prioritize documentation optimization before investing in skills, ensuring agents can navigate the platform effectively.