Orchestrating Data Quality with Airflow (www.astronomer.io)

A data team announced they embedded continuous, declarative data-quality testing directly into their Airflow pipelines by building reusable “test” task groups and nesting them inside existing pipeline templates (ETL, CreateTable, etc.). The motivation: data consumers were finding issues before engineers, eroding trust. By making tests part of the standard developer workflow, quality checks run every time data updates, making testing proactive, visible, and easier to adopt across ingests, models, and delivery layers. Technically, tests are specified in YAML frontmatter and rendered into SQL Column/TableCheckOperators with small wrappers that log metadata and failed records into the warehouse. The same task group can be invoked with different Airflow trigger rules to implement blocking (hard) vs non-blocking (soft) failures, and each check records an owner and severity so issues are triaged and surfaced via “Hygienies” dashboards and alerts. This approach turns testing into configuration, enforces data contracts, and enables shared ownership. Next steps include exploring self-healing, quantifying data health over time, and comparing metrics across runs so teams can measure improvements in trust and reliability rather than just point-in-time pass/fail.