3D and 4D World Modeling: A Survey (arxiv.org)

A new survey paper delivers the first comprehensive review explicitly focused on 3D and 4D world modeling and generation, addressing a fragmented literature and the lack of a standard definition for “world models.” The authors define precise terminology and introduce a structured taxonomy that separates approaches into video-based (VideoGen), occupancy-based (OccGen), and LiDAR-based (LiDARGen) methods. They catalog the major native 3D/4D representations—RGB-D imagery, occupancy grids, and LiDAR point clouds—summarize available datasets and evaluation metrics tailored to 3D/4D tasks, and provide a systematic literature index and demos to serve as a community resource. This survey is significant because it reframes world modeling beyond 2D image/video generative work and pushes the community toward unified evaluation, reproducibility, and clearer comparisons across methods. Key technical implications include the need for metrics and benchmarks that capture geometric and temporal coherence in 3D/4D outputs, best practices for fusing multi-sensor data, and categorization that clarifies tradeoffs between dense occupancy representations and sparse point-based LiDAR modeling. The paper also highlights practical applications (robotics, autonomous driving, AR/VR), open challenges—scaling, temporally-consistent generation, and standardization—and outlines promising research directions to accelerate robust, large-scale scene modeling.