ActCam: Zero-Shot Joint Camera and 3D Motion Control for Video Generation (arxiv.org)

A new zero-shot video generation method called ActCam has been unveiled, enabling impressive control over both actor motion and camera dynamics without requiring any additional training. By leveraging pretrained image-to-video diffusion models, ActCam transfers character motion from a reference video into a new scene while simultaneously managing camera parameters for each frame. This innovative approach involves conditioning on pose and sparse depth for scene structure, followed by refined high-frequency detail generation through pose-only guidance. ActCam's significance lies in its ability to combine joint motion and camera control effectively, yielding improved adherence to camera dynamics and enhanced motion fidelity compared to previous methods. The technique has demonstrated notable advantages, particularly in handling challenging viewpoint changes, as evaluated across multiple benchmarks. Such advancements could have profound implications for filmmakers and content creators in the AI/ML community, offering a powerful tool for generating high-quality video content with nuanced cinematographic control, thus transforming the landscape of automated video production.