Anyblox: One data encoding to rule them all (github.com)

AnyBlox is a new framework that lets datasets “decode themselves” by bundling lightweight WebAssembly (WASM) decoders with the data. Instead of each data system implementing readers for Parquet, ORC, JSON or new experimental formats, AnyBlox requires only a single host reader that runs a dataset-provided WASM decoder (one exported function) inside a sandbox. The framework exposes shared memory between host and decoder so decoding is fast and memory-efficient, while sandboxing provides security and portability across platforms. Integrations into existing engines take only a few hundred lines of code, immediately enabling support for cutting‑edge formats (e.g., Vortex), domain-specific formats (e.g., ROOT) or instance-optimized encodings tailored to particular datasets. For the AI/ML community this pattern is significant because it unblocks innovation in data encodings and compression: researchers and producers can invent new, highly efficient formats and make them instantly usable without changing downstream systems. The model reduces maintenance burden, speeds adoption of experiment-driven encodings, and preserves performance through shared-memory WASM execution. AnyBlox’s design emphasizes portability, security, performance and extensibility, and its paper won Best Paper at VLDB 2025, signaling strong academic validation for a practical approach that could reshape how ML pipelines consume diverse and evolving data formats.