Saber: Scaling Zero-Shot Reference-to-Video Generation (franciszzj.github.io)

Saber is an innovative zero-shot reference-to-video (R2V) generation framework designed to synthesize videos based on text prompts while maintaining the identity of subjects from reference images. Traditional R2V methodologies rely heavily on expensive and limited triplet datasets comprising reference images, videos, and text prompts, which hinders scalability and generalization. Saber circumvents these challenges by exclusively utilizing video-text pairs during training. It employs a masked training approach where randomly selected and partially masked video frames serve as dynamic substitutes for reference images, enabling the model to learn identity-consistent representations without the need for explicit R2V data. This breakthrough is significant for the AI/ML community as it demonstrates strong zero-shot generalization and scalability, outpacing models trained on traditional R2V datasets. Saber incorporates a tailored attention mechanism that focuses on relevant reference features while mitigating common copy-paste artifacts through spatial mask augmentations. Its architecture effectively integrates multiple reference images and views, allowing for sophisticated multi-subject customization in generated videos. This not only streamlines video generation processes but also opens new avenues for real-world applications in content creation, entertainment, and virtual environments where efficient adaptability and diversity in subjects are paramount.