SkillsBench: Benchmarking how well agent skills work across diverse tasks (arxiv.org)

A recent study introduces SkillsBench, a benchmarking framework designed to evaluate the effectiveness of agent skills—structured packages of procedural knowledge used to enhance large language model (LLM) agents. The research, involving 86 tasks across 11 diverse domains, assesses the performance of LLM agents under three different scenarios: without skills, with curated skills, and with self-generated skills. The findings reveal that curated skills improve average task pass rates by 16.2 percentage points, with significant variations depending on the domain, such as a 51.9 percentage point increase in healthcare. However, self-generated skills were found to be ineffective on average, indicating that models struggle to independently produce useful procedural knowledge. This research underscores crucial implications for the AI/ML community, particularly in refining how agent skills are developed and implemented. The results emphasize the necessity of curated knowledge over self-generated alternatives, as 16 out of 84 tasks even showed decreased performance with self-created skills. Additionally, the study suggests that focused skills—incorporating just 2-3 modules—can outperform more extensive documentation, and smaller models utilizing skills may rival larger models that operate without them. This highlights a potential avenue for optimizing agent performance, particularly in resource-constrained environments.