The SQL ran fine. 58% of the answers were wrong (AI agents on a real warehouse) (github.com)

A new open-source tool called “semlayer” automates the creation of a semantic layer for data warehouses, significantly improving AI agents' ability to understand and interact with complex datasets. By analyzing databases, semlayer infers relationships, identifies hidden business rules, and expresses this information in a structured, accessible format, eliminating the need for manual YAML configuration typically required by existing tools like dbt or LookML. This innovation aims to enhance the accuracy of AI responses to business queries, demonstrating marked improvement in performance on conventional benchmarks. In tests, AI agents using the inferred semantic layer correctly answered 87% of business questions, vastly outperforming the 42% accuracy achieved directly from raw schemas. The tool also features SQL linting capabilities that further boost accuracy by catching common errors like incorrect aggregations or missing filters. This development is particularly significant for organizations relying on messy, complex warehouses since it streamlines the AI integration process, facilitates smoother data operations, and allows for ongoing adaptation to schema changes without losing valuable insights—ultimately raising the standard for AI's utility in business decision-making.