Rogue: Open-source AI agent evaluation framework (github.com)

Rogue is an open-source framework for automated evaluation of AI agents that runs a dynamic EvaluatorAgent against your target using Google's A2A protocol to test policy compliance, reliability, and task correctness. It generates scenarios from a provided "business context," runs live agent‑to‑agent conversations, and produces Markdown reports with pass/fail metrics, findings and recommendations—making it useful for safety, prompt‑injection checks, and CI/CD integration. The project emphasizes observability (live chat monitoring), extensible testing (deep‑test mode), and multi‑model judging, so teams can systematically validate agent behavior before deployment. Technically, Rogue uses a client–server architecture: a central Rogue Server hosts evaluation logic while multiple clients connect—TUI (Go + Bubble Tea), Gradio Web UI, or a non‑interactive CLI for automated pipelines. It requires Python 3.10+, an LLM API key (OpenAI, Google/Gemini, Anthropic supported via LiteLLM format), and can be installed with uvx or pip; example agents (a T‑shirt store) and uvx commands simplify getting started. CLI options let you specify evaluated‑agent URL/auth, judge LLM/model and API key, business context file, scenario file, and output report. Hosted on GitHub under the Elastic license, Rogue is designed for integration into development workflows and invites community contributions to extend its scenario and safety checks.