Show HN: Largest open-source multimodal AI dataset (e-mm1.github.io)

A research team released the largest open-source multimodal dataset that explicitly links five modalities—text (captions), images, video, audio, and point clouds—into matching quintuples. The release includes a >100M automatically assembled pre-training pool (built by matching ~6.7M captions to top-16 candidates per modality using modality-specific embeddings: EvaCLIP for images/video, CLAP for audio, Uni3D for point clouds), a 1M human-rated post-training subset curated with hierarchical clustering and graph-based sampling to reduce bias and overlap, and a 3.5K consensus-evaluated zero-shot benchmark for audio↔point-cloud retrieval (each item vetted by five annotators). The team applied integrity checks, NSFW filtering, license reporting, and leakage controls to improve data health. This dataset matters because it lowers a major barrier to truly multimodal AI—scale and cross-modality alignment across perceptual and 3D modalities—enabling better cross-modal retrieval, multimodal GenAI, context-aware LLMs, and “PhysicalAI” research. The authors also provide a baseline joint-embedding model trained with contrastive pretraining on the automated pairs and fine-tuned on the 1M rated subset, showing strong retrieval and zero-shot performance while leaving clear headroom (e.g., full-token attention, quality-weighted objectives, augmenting ratings). All partitions, precomputed embeddings, baseline code, and configs are available on GitHub to accelerate replication and further research.