Observability Design for the AI Era – App, Infra, CI, LLM (Part 1) (ryantsuji.dev)

Ryan, CTO at airCloset, presents a new framework for observability in the age of AI, emphasizing the need to make production logs usable for AI models. Building upon insights from a previous series on creating a semantically searchable codebase, he stresses that raw log data can overwhelm AI systems, leading to confusion and inefficiency. As a solution, he proposes a structured observability stack divided into four axes—application, infrastructure, CI, and LLM—each tailored to specific queries that AI needs to address. This approach allows for meaningful insights without information overload, ensuring that AI can accurately interpret dynamic production environments. The significance of this framework lies in its potential to enhance how AI interacts with observability data, making it more effective in real-time decision-making processes. For instance, by instrumenting CI logs into Grafana Loki and integrating various metrics through OpenTelemetry, airCloset can quickly garner insights about production performance and CI failures, enabling proactive responses. Additionally, distinct tracking methods for AI model usage, such as Gemini and Claude Code, delineate between real-time cost tracking and historical usage aggregation, which fosters better resource management and cost optimization for organizations leveraging AI technology.