Learnings from 4 months of Image-Video VAE experiments (www.linum.ai)

Over the past four months, a team has been developing an Image-Video Variational Autoencoder (VAE), grappling with numerous technical challenges, including instability in co-training and unexpected failures during the reconstruction process. Their findings highlighted that the pursuit of high reconstruction quality may not be as crucial as initially thought, leading them to incorporate various strategies to achieve a balance between image and video processing. A notable insight was the realization that the size differences between video and image data skewed loss calculations, causing the model to prioritize video reconstruction over images. To address this, they normalized the loss relative to a fixed reference shape, allowing for more consistent training across different modalities. The significance of this work lies in its contribution to optimizing generative models, particularly in text-to-video applications where efficient latent space compression is paramount. By detailing their experimentation with VAEs, including the replacement of complex Group Norms with Self-Modulating Convolutions to address visual artifacts, the team provides valuable lessons for the AI/ML community. Ultimately, while the team has transitioned to using an existing VAE model (Wan 2.1), their insights emphasize the importance of robust training techniques and the nuanced understanding that better reconstruction does not necessarily translate to superior generation capabilities.