Training Text-to-Image Models Without a VAE (www.linum.ai)

A new architectural advancement in the AI field has emerged with the introduction of the Pyramid-JiT (P-JiT), a decoder-only pixel space model that improves text-to-image generation by eliminating the Variational Autoencoder (VAE). This innovative architecture compresses input tokens more aggressively, drastically reducing training and inference costs while achieving significant performance gains. Notably, P-JiT outperforms previous models, such as Linum v2's FD-DINOv2, by requiring 11.3 times fewer training samples and 4.3 times fewer GPU hours, while also handling images at four times the pixel resolution. P-JiT's architectural design revolves around predicting target images at multiple resolutions, optimizing the complexity of attention mechanisms and compressing training data. This advancement not only enhances the efficiency of video generation tools but also builds on lessons learned from language models that successfully transitioned from encoder-decoder to decoder-only structures. By sharing their code and model weights under an Apache 2.0 license, the researchers aim to inspire further exploration of efficient training methods within the broader AI/ML community, paving the way toward more accessible and affordable animation and video generation capabilities.