Who invented deep residual learning? (people.idsia.ch)

Sepp Hochreiter first introduced the core idea behind deep residual learning in a 1991 diploma thesis: identity self-connections with weight 1.0 in recurrent neural units that guaranteed "constant error flow" and solved the vanishing/exploding gradient problem. That principle became central to LSTM cores (1995–1997), which used constant-error carousels (CECs) and later gated variants (1999) whose forget gates were initialized open (1.0). Unfolding recurrent models into time-expanded feedforward networks (mid-2000s) made these recurrent residuals functionally equivalent to feedforward residual paths. In May 2015, Highway Networks formally ported gated residuals to deep feedforward nets (g(x)x + t(x)h(x)), and seven months later ResNet popularized the ungated, always-open variant—effectively an open-gated Highway Net—which propelled training of hundreds-layer nets and won ImageNet 2015. The technical significance is that true “residual” connections are identity paths with weight 1.0 (not arbitrary skips): they preserve gradients across many layers or time steps, enabling very deep architectures. Gates let networks selectively disable residual behavior but are typically initialized to keep identity flow. This lineage explains why ResNets, Highway Nets, LSTMs and even modern Transformers all rely on identity-like shortcuts to scale depth reliably. In short: Hochreiter coined the foundational solution in 1991; subsequent work adapted, gated, unfolded and popularized it for today's deep learning renaissance.