LeJEPA: Provable and Scalable Self-Supervised Learning Without the Heuristics (arxiv.org)

LeJEPA is a new, theory-driven formulation of Joint-Embedding Predictive Architectures (JEPAs) that replaces the ad-hoc heuristics common in self‑supervised learning with a provable, scalable objective. The authors show theoretically that the optimal embedding distribution for minimizing downstream prediction risk is an isotropic Gaussian, and introduce Sketched Isotropic Gaussian Regularization (SIGReg) — a lightweight, sketch-based regularizer that steers embeddings toward that ideal. Combining SIGReg with the standard JEPA predictive loss yields LeJEPA, which the paper claims is heuristics-free (no stop-gradient, teacher-student models, or complex schedulers), uses a single trade-off hyperparameter, and can be implemented in roughly 50 lines of code. Technically notable is LeJEPA’s computational profile and empirical breadth: SIGReg gives linear time and memory complexity via sketching, making the method friendly to large models and distributed training, while empirical tests span 10+ datasets and 60+ architectures (ResNets, ViTs, ConvNets). In practice the approach is stable across hyperparameters and domains — for example, ImageNet‑1k pretraining plus linear evaluation with a frozen ViT‑H/14 backbone achieves 79% — suggesting LeJEPA could simplify and standardize JEPA-style SSL pipelines, improve reproducibility, and scale self-supervised pretraining without the usual engineering tricks.