Skar – turn a captured AI agent trace into a pytest regression test (github.com)

Skar, a new tool for AI development, enables teams to transform captured traces of AI agent interactions into committed pytest regression tests. This process is especially significant for engineers who wrap large language models (LLMs) into tool-using agents, as it allows for the locking of specific AI behaviors as tests within their codebase. Skar is designed to assist in scenarios where a custom agent encounters issues, providing a means to ensure that such problems do not recur by capturing the trace and converting it into a test easily runnable in continuous integration setups. The tool consists of two main components: the CLI and MCP server for trace management and validation, and the Python runtime for capturing agent runs. Engineers can utilize features like trace validation, inspection, and regression test generation, all structured around a specific trace schema. While Skar excels at capturing and testing tool-using behaviors, it intentionally does not address issues like changes in LLM behavior or tool result regressions, as these are outside its design scope. As a result, Skar fills a niche in the AI/ML testing landscape, providing a reliable way to handle regression testing in custom-built AI agents while maintaining control over the testing environment.