Show HN: Rust Backed Fast Dataloader (github.com)

A new Rust-backed fast dataloader named Ferroload has been introduced, featuring a multimodal dataset format designed for efficient machine learning (ML) training. Ferroload employs a pure-Rust implementation with Python bindings, enabling it to handle sharded tar data with a columnar DuckDB-queryable Parquet index. This allows for rapid streaming of datasets from local disks or cloud storage, and its parallel decoding minimizes processing overhead. Users can easily write, query, and train on datasets with a straightforward API, supporting various cloud platforms including S3, GCS, and Azure. The significance of Ferroload lies in its performance benchmarks, demonstrating substantial speed improvements over existing datasets like WebDataset and Hugging Face datasets. For instance, Ferroload achieved a 3.9× speed increase over WebDataset in loading times, particularly excelling at JPEG decoding due to its integration with optimized libraries like libjpeg-turbo. The software is designed for cloud-native applications, allowing remote datasets to be accessed and manipulated efficiently. Key features include deterministic reshuffling for training, image resize capabilities, and a CLI for catalog management—all of which contribute to making Ferroload a powerful tool for data-intensive ML projects.