OTelBench: AI struggles with simple SRE tasks (Opus 4.5 scores only 29%) (quesma.com)

Researchers have introduced OTelBench, an open-source benchmark that evaluates AI models' abilities to perform OpenTelemetry instrumentation—tasks crucial for debugging in microservices architecture. In testing 14 leading models across 23 tasks, the top performer, Claude Opus 4.5, achieved a mere 29% success rate, highlighting a significant gap in the capabilities of AI in handling real-world Site Reliability Engineering (SRE) tasks. This benchmark demonstrates that while AI excels in generating code, it struggles to effectively understand and implement the nuances of distributed tracing, which links distinct events generated across various services. The results underscore a critical limitation within current AI models: the failure to recognize and properly link separate user actions, often leading to conflated traces or malformed data. This shortcoming reveals that despite advancements in AI, frontline models still lack the necessary skills for comprehensive observability tasks required in modern software development environments, characterized by their complexity and multilinguistic frameworks. As the demand for reliable and efficient distributed tracing grows, OTelBench serves as a call to action for the AI/ML community to refine these models, suggesting a roadmap towards creating more robust systems in AI-assisted reliability engineering.