Building AI skills like checklists (blog.lawrencejones.dev)

incident.io has created an AI-powered skill to automate the analysis of their spending on AI inference, addressing the complexities of tracking multiple dimensions, such as spend by feature and anomalies. The skill operates like a runbook, allowing AI agents to execute specific processes that would typically require manual tracking and reporting. This automation is significant for the AI/ML community as it demonstrates a practical application of AI in optimizing operational costs, where the growth of AI inference expenses demands meticulous oversight. The process of developing this skill heavily incorporates the principles from Atul Gawande's "The Checklist Manifesto," emphasizing that executable documentation must be rigorously tested and iterated upon to ensure accuracy and reliability. An initial draft of the skill was refined through real-world testing and analysis, revealing insights that shaped its functionality and highlighted critical areas needing attention, such as tagging and data routing issues. By embedding a continuous dry-run loop practice and soliciting feedback from both human and AI agents, the skill not only evolves but increasingly aligns with actual operational needs, suggesting a shift towards more intelligent and adaptable AI systems in managing complex data environments.