How far are we from scaling up next-pixel prediction for image pretraining? (arxiv.org)

This paper studies how autoregressive next-pixel prediction—training transformers to predict pixels sequentially—scales as compute, model size, data, and resolution grow. Using IsoFlops profiles, the authors train a family of Transformers on 32×32 images across compute budgets up to ~7×10^19 FLOPs and evaluate three targets: next-pixel loss, ImageNet classification accuracy, and generative quality (Fréchet Distance). Key empirical findings are that optimal scaling is task-dependent: for fixed 32×32 resolution, setups that best serve generation require the dataset to grow 3–5× faster than those tuned for classification. As resolution increases, the optimal recipe shifts toward much faster growth in model size than data. Projecting trends, they identify compute—not data—as the main bottleneck and forecast that, if compute continues to grow ~4–5× per year, practical pixel-by-pixel image modeling at larger scales could be feasible within about five years. The significance for AI/ML is twofold: it challenges the common assumption that data scarcity is the primary limit and shows unified pixel-prediction pretraining can be competitive if compute and model-sizing are prioritized differently per task. Practically, the work provides concrete scaling prescriptions (IsoFlops experiments, task-conditioned data-vs-model trade-offs) that inform resource allocation, dataset collection, and architecture choices for generative and discriminative vision systems moving toward unified end-to-end models.