Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills (arxiv.org)

A new framework named Agentic Continuous Evaluation of Skills (ACES) has been introduced to enhance the way enterprise agent programs evaluate their skill capabilities, transitioning from mere prototypes to fully functional production-level tools. This innovative framework emphasizes the evaluation of reusable skills, tools, and workflow packages based on factual evidence rather than subjective descriptions. ACES conducts live trials comparing agents with and without specific skills, normalizing results into the Agent Trajectory Interchange Format (ATIF) and providing metrics on Skill Lift—the additional value a skill brings to task completion. Significantly, the ACES methodology reveals performance insights beyond what traditional scan-only evaluations can provide, as evidenced by a positive Skill Lift observed in 72.8% of cases among 947 scored examples. Key metrics reflect substantial improvements in skill execution and efficiency, offering crucial data on agent behavior, workflow adherence, and tool utilization. The open-source nature of ACES, accessible via NVIDIA SkillEvaluator, promotes wider adoption and adaptation within the AI/ML community, encouraging ongoing development and validation of skills that drive effective enterprise solutions.