Are Video Models Ready as Zero-Shot Reasoners? (video-cof.github.io)

Researchers ran the first systematic evaluation of a leading video generation model, Veo-3, to test whether modern video models can act as zero-shot visual reasoners. They built MME-COF, a compact benchmark and protocol for Chain-of-Frame (CoF) reasoning, and assessed Veo-3 across 12 dimensions—spatial, geometric, physical, temporal, embodied logic, and more—cataloging representative successes and characteristic failures. Empirically, Veo-3 can synthesize locally coherent, short-horizon trace animations in simple, low-branching scenarios and shows promising short-horizon spatial coherence, fine-grained grounding, and locally consistent dynamics. However, the study finds clear limitations: Veo-3 struggles with long-horizon causal planning, strict geometric constraints, abstract logical rules, and reliably executing multi-step, rule-grounded sequences. These failure modes mean current video models are not yet dependable as standalone zero-shot reasoners, especially for tasks requiring extended temporal planning or provable constraint satisfaction. Technically, the work emphasizes the need for standardized benchmarks like MME-COF to reveal nuanced behavior beyond visual fidelity and suggests a practical path forward: using video models as complementary perceptual or imagination engines paired with dedicated symbolic or chain-of-thought reasoners, while future research targets improved long-horizon consistency and explicit rule grounding.