Beating a text-to-SQL benchmark: can you get better than plain Claude? (motley.ai)

Motley has announced a significant advancement in text-to-SQL capabilities by utilizing its open-source semantic layer, SLayer, and the Claude SDK. An agent equipped with this combination achieved a remarkable 75.3% pass rate on the BIRD-INTERACT benchmark, far surpassing the previous best of 36.33% recorded on the Lite leaderboard. The standout feature was not solely the addition of the semantic layer but rather the use of the Claude SDK, which lifted the performance of a raw-SQL agent from 71.7% to the higher benchmark. This development is crucial for the AI/ML community as it highlights not only the potential of semantic layers in enhancing model capabilities but also emphasizes the importance of agent architectures in achieving high performance in complex tasks like text-to-SQL. The experiment showed the need for addressing mislabeled benchmark answers, where an annotation agent was used to differentiate correct outputs from false negatives. The insights gained from this benchmark will help refine SLayer and further enhance its ability to process and generate SQL queries in real-world scenarios, ultimately driving better AI tools for data interaction.