Training and Deploying Production Cyber Monitors on Kimi K3 (www.goodfire.com)

Researchers have successfully developed and deployed advanced cybersecurity monitors for the Kimi K3 and GLM 5.3 AI models that significantly outperform traditional large language model (LLM) judges at a fraction of the cost. These probe-based monitors offer up to 93% recall with just a 5.5% false positive rate, while costing approximately $200 per million exchanges—about 50 times less than using a conventional LLM in real-time monitoring. This innovation is crucial as it enhances real-time monitoring capabilities, enabling the identification and blocking of harmful AI actions before they can be executed, a necessity given the similar patterns between benign and malicious behaviors in cybersecurity contexts. The new monitoring architecture utilizes a cascade system where a lightweight probe assesses internal model activations and flags only suspicious interactions for further review by an LLM judge. This efficient design minimizes computational costs and latency, ensuring that monitoring does not slow down AI operations. Performance evaluations show that the implementation not only maintains throughput but dramatically reduces additional latency from 160 ms to mere 2 ms per turn, making it a viable solution for deploying AI models in sensitive environments where safety is paramount. These advancements offer a practical safety framework for developers utilizing open models in agentic settings.