How to make scientific data archives AI-ready (www.earthmover.io)

A new approach has been proposed for transforming scientific data archives into AI-ready systems, addressing a significant limitation that hinders the utility of AI tools in scientific research. The existing challenge lies in the fact that scientific data is predominantly stored in files, which are not optimized for easy access by AI agents or foundation models. This results in a cumbersome process for researchers who must navigate vast numbers of opaque files to find relevant datasets. By shifting to object storage, indexing data chunks instead of files, and co-locating metadata alongside the data, this system aims to provide a more structured and efficient way to access and utilize massive scientific datasets. The proposed method leverages scalable storage solutions and cloud-optimized formats like Icechunk and Zarr, which offer significant efficiency improvements—such as NASA's reported 100x speedup in query responses. This new architecture not only enhances access for AI model training but also ensures consistent metadata, accommodating concurrent usage by both human and AI users. By cataloging datasets rather than individual files, the system simplifies querying while allowing for fine-grained access control, thus making it a more effective resource for both scientific inquiry and AI applications. Ultimately, this evolution in data archiving methodology stands to bolster the overall productivity of scientific research, enabling a more synergistic relationship between data availability and AI capabilities.