ShotPlan: Cinematic Video Generation with Learnable Planning Token (pensioner-11.github.io)

ShotPlan is a groundbreaking system designed for cinematic video generation, utilizing the Wan2.1-T2V-14B model while introducing minimal modifications through the use of learnable planning tokens. By simply appending these tokens at fractional temporal positions, ShotPlan effectively enables precise control over video transitions, such as hard cuts and fades, without altering the model's core architecture. Each transition type is assigned a unique embedding that allows for flexible timing and smoother video production, demonstrated through innovative techniques like Fractional Rotational Positional Encoding (FRoPE), which ensures the video maintains a high degree of coherence and visual fidelity. This advancement holds significant implications for the AI/ML community, particularly in creative applications focused on video generation and editing. ShotPlan has been tested against existing benchmarks, significantly outperforming competitors in transition accuracy, narrative coherence, and character consistency, with transition deviation reduced to under one frame. By enhancing the precision of generated video content, ShotPlan paves the way for more sophisticated AI-driven tools in the film and content creation industries, ushering in a new era where machine learning plays an integral role in cinematic storytelling.