Dust: Pretraining Transformers Without Backpropagation (qlabs.sh)

Researchers have introduced Dust, a novel zeroth-order optimization method for pretraining transformer language models that operates without backpropagation. Instead of the traditional backpropagation method, which requires the calculation of gradients through differentiable structures, Dust perturbs the activations of tokens independently, effectively creating a "virtual population." This allows for simultaneous evaluation of numerous perturbations in a single forward pass, making it significantly more efficient—by factors of up to 10,000—than existing weight-space evolution strategies like EGGROLL, especially when dealing with large datasets. The significance of Dust lies in its potential to operate in a compute-rich environment, where it not only competes with but can also exceed backpropagation's efficiency as the population of perturbed tokens increases. Contrary to the common belief that larger models diminish efficiency, Dust demonstrates that larger networks can utilize greater populations effectively, leading to improved gradient estimates that align closely with those produced by backpropagation. This research challenges existing paradigms in machine learning by suggesting that reliance on differentiable methods may limit architectural exploration, paving the way for a new approach that prioritizes brute-force computation over traditional inductive biases.