Optimizing Iceberg Compaction: Why We Built an Embedded Engine in Rust (risingwave.com)

RisingWave has introduced an embedded compaction engine in Rust to optimize Apache Iceberg's performance. As Iceberg is widely used for its features like ACID transactions and schema evolution, the accumulation of snapshots and delete files can lead to significant performance overhead, particularly in streaming databases. To combat this “small file” problem and metadata bloat, RisingWave's new architecture integrates DataFusion directly into its system, allowing for efficient data compaction that includes strategies like snapshot expiration and file merging. This innovation holds great significance for the AI/ML community, especially for applications reliant on real-time data processing. The new design dramatically enhances performance, yielding a benchmark speedup of 5.5 times over traditional Apache Spark configurations, while also proving to be more memory-efficient. RisingWave’s dual write modes—Merge-on-Read and Copy-on-Write—cater to both ingestion speed and read performance, adapting to various user needs without the complexity of managing additional infrastructure. This unified approach not only streamlines maintenance but also positions RisingWave as a robust, cost-effective solution for handling high-velocity streaming data.