Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers (arxiv.org)

Researchers have introduced a novel method for enhancing Vision Transformers (ViTs) by pretraining them on procedurally generated data that lacks visual or semantic content. This innovative approach utilizes simple algorithms, such as formal grammars, to create abstract data, enabling the ViTs to internalize generic computational biases before being trained on standard image datasets. Notably, this “warm-up” phase circumvents the traditional visual patch embedding mechanisms and leads to significant improvements in data efficiency and model performance. The results are compelling: when ViTs are pretrained with only 1% of their training budget allocated to this procedural data, they achieve an increase of over 1.7% in accuracy on the ImageNet-1k benchmark, translating to a performance boost equivalent to using 28% of the traditional dataset. This research opens up exciting possibilities for future AI training strategies by highlighting the potential of domain-agnostic pretraining, which could lead to more data-efficient models that generalize better across various applications.