Zip Files as (Simple) Key-Value Stores (benjamincongdon.me)

A recent exploration into distributing key-value datasets revealed that zip files can serve as efficient, low-dependency key-value stores for read-only workloads in constrained memory environments. The scenario involved managing around 10k string keys with opaque value blobs ranging from 100B to 1MB, where memory constraints prohibited loading all data at once. The author tested various approaches, including SQLite and a custom file format, ultimately finding that zip files outperformed SQLite for smaller blob sizes due to their efficient directory structure that allows for direct access to individual files without requiring full decompression. This insight is significant for the AI/ML community as it highlights a simple yet effective alternative for data storage, particularly where memory resources are limited. Zip files not only minimize overhead associated with more complex database systems but also ensure that key access patterns remain efficient. With the added benefit of wide language support for zip file handling, this approach could streamline certain workflows in data-heavy AI applications, proving that sometimes straightforward solutions can deliver competitive performance without the complexity of traditional database engines.