Making Gaussian Splats Searchable (spatialview.io)

A new innovative approach has been introduced that enhances the searchability of high-fidelity 3D models by integrating Large Multimodal Models (LMMs) with traditional 3D reconstruction techniques like Gaussian Splatting and Neural Radiance Fields (NeRFs). This system addresses the semantic gap in 3D models, which often lack meaningful indexing for efficient querying. By automating the semantic interpretation of visual data, users can now conduct natural language searches directly in 3D environments, transforming passive visual records into actionable, machine-readable assets. The process begins with recovering camera positions using methods like Structure from Motion (SfM) and generating 3D models tied to specific locations. It employs vector embeddings to create a searchable index of imagery, allowing for rapid identification of relevant frames in response to user queries. This capability extends to identifying unique object instances and their spatial relationships, ultimately mapping 2D coordinates to 3D space. As a result, users can perform complex queries efficiently, such as locating equipment conditions, leading to significant improvements in operational workflows, documentation, and inventory management. This advancement signifies a significant step towards creating dynamic digital twins that are not only visually accurate but also programmatically accessible and practical for various applications in the AI/ML community.