Anomaly detection for alerting: statistics, LLMs and decision models compared (anomaly-detection.coroot.com)

Coroot has conducted a thorough comparison of various anomaly detection methods for alerting, including classic statistical models, large language models (LLMs), and decision models inspired by Jev's framework. This exploration was prompted by ongoing discussions about the necessity of Service Level Objectives (SLOs) in a landscape where AI could potentially handle the alerting process with minimal configuration. The study specifically addresses a common scenario faced by on-call engineers—deciding if a web service's p99 latency warrants a late-night alert—by evaluating each method's performance based on real-time data. The results are significant for the AI/ML community, as they highlight that Coroot's multi-window error budget burn rate approach consistently outperforms other models, including advanced LLMs. By scoring each method on its ability to correctly predict when an alert should be raised or silenced based on human judgment, the findings underscore the complexity of effective anomaly detection in dynamic environments. This comparison not only advances the understanding of alerting mechanisms but also opens avenues for more robust AI systems that can integrate decision-making with real-world operational criteria.