We gave terabytes of CI logs to an LLM (www.mendral.com)

A recent experiment revealed the impressive capability of a machine learning agent that can swiftly analyze vast terabytes of Continuous Integration (CI) logs to identify issues. By autonomously crafting SQL queries, the agent was able to trace a flaky test back to a dependency update in mere seconds, parsing through 1.5 billion log lines weekly. The underlying data architecture employs ClickHouse, enabling rapid query performance and efficient storage through a compressive columnar format that dramatically reduces the size of stored logs—yielding a 35:1 compression ratio—while maintaining rich context through 48 metadata columns per log line. This advancement holds significant implications for the AI/ML community, particularly in how machine learning applications can enhance software development and debugging processes. The agent's design allows for broad query flexibility beyond predefined libraries, enabling it to address unexpected failures and issues with agility. By continuously polling GitHub’s API and intelligently managing data ingestion rates to maintain freshness, the system avoids the pitfalls of stale data that could hinder real-time debugging efforts. Leveraging durable execution, it adeptly handles bursts of CI activity without downtime, indicating a promising direction for future AI-driven tools aimed at optimizing developer workflows and improving operational efficiency in software engineering.