One AI Model Creates a Physical Intuition of Its Environment (www.quantamagazine.org)

Meta’s Video Joint Embedding Predictive Architecture (V-JEPA) is a video-pretraining model that learns a rudimentary “physical intuition” of the world from raw video—without built-in physics rules—by predicting compressed, high-level (latent) representations instead of pixels. During pretraining the system masks the same regions across frames, feeds masked frames into encoder 1 and unmasked frames into encoder 2, and trains a predictor to map encoder-1 latents to encoder-2 latents. This focus on abstracted features lets the model ignore irrelevant pixel noise (like leaf motion) and concentrate on objects, motion, occlusion and causality, enabling much more sample-efficient adaptation to downstream tasks. V-JEPA’s behavior mirrors infant-like surprise: when future frames violate learned physical expectations (e.g., an occluded ball fails to reappear), prediction error spikes. On the IntPhys benchmark it scored ~98% vs near-chance for a pixel-space baseline. Meta released V-JEPA 2 (1.2B parameters, pretrained on 22M videos) and demonstrated robot fine-tuning using only ~60 hours of robot data to plan actions. Important limits remain: V-JEPA lacks calibrated uncertainty, only retains a few seconds of memory, and both it and V-JEPA 2 struggle on a tougher IntPhys 2 benchmark. Still, the work is significant for AI/ML because it shows large-scale latent-based video models can induce intuitive physics from passive observation and be efficiently repurposed for perception and control.