Show HN: Apache DataFusion Support for DuckDB's DuckLake (ducklake.select)

Hotdata has successfully integrated DuckLake into Apache DataFusion, enabling the query engine to utilize a lakehouse format for managing metadata and snapshots of Parquet files stored in object storage. This implementation is significant for the AI/ML community as it allows for rapid creation and management of isolated databases tailored to AI applications, supporting high-volume, latency-sensitive queries while efficiently handling millions of ephemeral databases. By open-sourcing this advancement within the Apache ecosystem, Hotdata encourages further collaboration and innovation in data management. Technically, DuckLake's architecture stands out by using a relational database for metadata operations instead of relying on multiple object-store file requests, streamlining metadata lookups. Benchmark tests revealed that DuckLake-backed tables performed slightly better than direct Parquet access, attributed to reduced planning overhead. The integration enables features like schema evolution and catalog metadata management without sacrificing performance, making DuckLake a compelling addition to DataFusion. This move not only enhances DataFusion's capabilities but also lays the groundwork for future developments in lakehouse technology within the community.