AI tooling challenges in Data Engineering (philippeoger.com)

Recent discussions highlight significant challenges when integrating AI into data engineering, particularly regarding the visibility of data models. Unlike backend engineering, where data structures are well-defined in the codebase, data engineering often struggles with schemas that are external to repositories. This disconnect can lead to errors when AI tools attempt to generate or fix ETL processes, as they lack direct access to the actual structure and definitions of data tables. To address this issue, a new practice has been proposed: implementing project-level schema definitions through dedicated files such as SCHEMAS.md. This file would contain precise data definitions relevant to specific ETL tasks, allowing AI models to reference accurate schemas and thus reduce hallucinations and errors during processing. By adopting this method, the AI can make informed decisions based on comprehensive and contextual information, ultimately enhancing accuracy in data transformations and leading to improved best practices as AI adoption in data engineering grows.