🤖 AI Summary
An experiment compared three sorting approaches—Java’s single-threaded Collections.sort (copying the list), Java’s multi-threaded List#stream().parallel().sorted().toList(), and DuckDB’s ORDER BY over a single-column table with results pulled back via Arrow—on random 32-bit integers and 4–16 char strings. The author ran the tests locally on a Mac Studio (Apple M2 Max, 32 GB), DuckDB 1.4.1, JDK 25, and shared the source on GitHub. The setup intentionally exercised DuckDB’s redesigned, multi-threaded sort against JDK’s built-ins; the author notes these are informal benchmarks, not rigorous measurements.
Results show DuckDB is competitive and often faster at large scales: for integers, DuckDB beat Java’s parallel sort by roughly 2x at high cardinalities (10M: DuckDB 143 ms vs JDK parallel 281 ms; 100M: 1,827 ms vs 3,911 ms; single-threaded JDK blew past 30s at 100M). For strings the gap is much smaller—at 10M elements DuckDB was 538 ms vs JDK parallel 578 ms—despite DuckDB incurring JNI/Arrow export and extra string recreation overhead (and more garbage). The takeaway: modern vectorized, multi-threaded DB engines like DuckDB can outperform or match JVM-native parallel sorts on large, real-world workloads, suggesting data-engine primitives (Arrow, SIMD-friendly execution, better concurrency) are worth considering for heavy in-memory sorting in ML/data pipelines.
Loading comments...
login to comment
loading comments...
no comments yet