Rectified Linear Units Improve Restricted Boltzmann Machines (2010) [pdf] (www.cs.toronto.edu)

Hinton and Nair (2010) show that replacing binary stochastic hidden units in Restricted Boltzmann Machines (RBMs) with rectified linear units (implemented as a Noisy ReLU approximation of infinitely many shared-weight binary copies) yields better learned features and improved downstream recognition. They keep the standard RBM learning framework (Contrastive Divergence) but reinterpret a stack of offset binary copies as a smooth, integer-valued unit whose expected activation closely follows log(1+exp(x)) and can be efficiently approximated by max(0, x + Gaussian noise). Empirically, RBMs with these NReLUs produced stronger features for object recognition on Jittered‑Cluttered NORB and for face verification on LFW, and gave better discriminative performance after backprop fine‑tuning. Technically, the paper provides a probabilistic grounding for ReLU-style nonlinearities in a generative model: stepped-sigmoid units (bias offsets −0.5, −1.5, …) sum to the smoothed rectifier, Gaussian visible units handle real-valued pixels, and the deterministic y = max(0,x) is used for backprop (gradient 0 for x≤0, 1 for x>0). A key insight is intensity equivariance — ReLUs preserve relative intensity information across layers (unlike binary units), which stabilizes feature responses under global lighting/scaling changes. The work helped justify using ReLUs in deep architectures and bridged discriminative success with a clear generative interpretation.