Finding a Needle in the Haystack: Querying Physical AI Data with Daft (www.eventual.ai)

Daft has introduced a groundbreaking capability for querying physical AI data, demonstrated through its application on Apple's EgoDex dataset, which includes varied hand poses and video footage of tabletop tasks. By allowing users to search through video data with natural language queries—such as "find every clip where a writing-gripped hand lifts chopsticks"—Daft effectively transforms the way researchers can navigate extensive, unlabeled multimodal datasets. This development signifies a major leap for the AI/ML community, particularly in robotics and autonomous systems, where the challenge of finding specific scenarios within vast amounts of unstructured data has been a persistent issue. The technical innovation lies in Daft's ability to combine frame-level semantic embeddings from Google's SigLIP-2 image encoder with computed geometric features derived from raw sensor data. Researchers can now capture complex hand movements and grips by processing both visual data and detailed wrist and finger positions, enabling sophisticated queries based on action states and geometric properties. This system not only facilitates targeted data audits and retraining but also enhances the ability to identify edge cases and nuanced behaviors within any dataset, addressing a critical data understanding problem facing robotics labs today.