RTFM: A Real-Time Frame Model (www.worldlabs.ai)

World Labs announced RTFM (Real-Time Frame Model), a research-preview generative World Model that produces interactive video in real time and can be explored like a persistent 3D world. RTFM is built around three principles—efficiency (runs interactive inference on a single H100 GPU), scalability (an end-to-end model that learns from large-scale video rather than relying on explicit 3D representations), and persistence (worlds don’t “forget” as you explore them). The system was engineered with architecture innovations, model distillation, and inference optimizations to deliver a practical preview of what compute-hungry World Models could become. Technically, RTFM is an autoregressive diffusion transformer operating on sequences of frames and acts as a learned renderer: input 2D views are converted into neural activations (a KV cache) that implicitly represent the scene, and attention-based decoding generates novel viewpoints consistent with lighting, reflections, and geometry seen during training. To achieve unbounded persistence without attending to an ever-growing context, RTFM attaches poses (position + orientation) to frames and uses "posed frames" as spatial memory—retrieving nearby frames via “context juggling” to build a local context when sampling new views. The approach sidesteps explicit 3D reconstruction while making world-scale, persistent generation tractable today and amenable to future scaling for dynamic interaction in media, robotics, and simulation.