Vanishing Gradients: Why Deep Networks Couldn't Learn

Gradient descent works. Take the gradient of the loss, step in the opposite direction, repeat. For a network with a few layers, this trains just fine. But there is a problem hiding in the depth. Make the network deep enough, ten layers, twenty, fifty, and the gradient signal that is supposed to reach the early layers fades to nothing. The first layers stop learning entirely. This is the vanishing gradient problem.

One product of many matrices#

Each layer of a feedforward network takes the previous layer's output, multiplies by a weight matrix, adds a bias, and applies an activation function:

$$h_k=f(W_k{h}_{k-1}+b_k)$$

To train the network, the error at the output has to propagate backward through every layer. When inputs and outputs are vectors, the derivative of one with respect to the other is a matrix, the Jacobian: each entry says how one output moves when you nudge one input. The chain rule gives the gradient at layer $m$ as the gradient at the output times a product of Jacobians, one per layer in between:

$$\frac{\mathit{\partial }L}{\mathit{\partial }{W}_m}=\frac{\mathit{\partial }L}{\mathit{\partial }{h}_L}·J_L{J}_{L-1}\cdots J_{m+1}·\left(\text{localterm}\right)$$

Each Jacobian factors as $J_k=D_k{W}_k$, where $D_k$ is a diagonal matrix holding one activation derivative per neuron. The product is the part that matters. If each factor shrinks the signal, the whole product shrinks, and with many factors it shrinks exponentially.

The sigmoid wall#

The original deep networks used the sigmoid activation, $\sigma \left(z\right)=\frac{1}{1+e^{-z}}$, a smooth curve that squashes everything into the range zero to one. Its derivative works out to ${\sigma }^{\prime }\left(z\right)=\sigma \left(z\right)(1-\sigma (z\left)\right)$: the output times one minus the output.

How large can that be? The output is some $p$ between zero and one, and the derivative is $p-p^2$. Maximize: the derivative with respect to $p$ is $1-2p$, which is zero at $p=\frac{1}{2}$, giving $\frac{1}{2}·\frac{1}{2}=\frac{1}{4}$. The sigmoid derivative is at most $0.25$. Everywhere. Always.

With standard initialization, the weight matrices have norms close to one, so the activation derivative is the dominant bottleneck; with large weights the product explodes instead, same disease, opposite sign. Each sigmoid layer lets through at most about a quarter of the gradient signal. Now compound that:

• five layers back, $(0.25{)}^5\approx {10}^{-3}$, a thousandth of the signal
• ten layers back, $(0.25{)}^{10}\approx {10}^{-6}$, one in a million
• twenty layers back, $(0.25{)}^{20}\approx {10}^{-12}$, a trillionth

Run the experiment and you can see it. Take a ten-layer network, eight neurons per layer, sigmoid everywhere, and plot the gradient magnitude of every neuron as a heatmap. The last few layers are bright. The middle is dim. The first layers are almost black. The early layers, the ones supposed to learn the most basic features of the input, receive essentially nothing.

This was not a surprise to the people in the field. Sepp Hochreiter identified the problem in his 1991 diploma thesis, analyzing how error signals decay exponentially in recurrent networks; he and Jürgen Schmidhuber later addressed the recurrent case with LSTMs, which maintain gradients through an additive path. Bengio, Simard, and Frasconi formalized the general problem in 1994. The practical consequence: for most of the 1990s and 2000s, networks had one or two hidden layers. Depth was considered impractical.

ReLU and careful initialization#

The first real dent came from changing the activation. ReLU outputs $x$ when $x$ is positive and zero otherwise, so its derivative is exactly one for active neurons and exactly zero for inactive ones. No squashing: an active neuron passes its gradient through at full strength. The same ten-layer heatmap with ReLU shows much more uniform gradients, no exponential fade. But it has black spots. On any given input about half the neurons are inactive, which is expected, but if a neuron's weights drift so that its input is negative for every training example, it goes dead, and in practice it rarely recovers. ReLU fixes the systematic shrinkage and introduces its own failure mode.

The second advance was initialization. A layer's pre-activation $z$ is a sum of $n$ terms, weight times input. For independent zero-mean weights and inputs, $\text{Var}\left(z\right)=n·\text{Var}\left(W\right)·\text{Var}\left(h\right)$. For the signal to neither grow nor shrink, you need $\text{Var}\left(W\right)=\frac{1}{n}$. Glorot and Bengio's 2010 Xavier initialization balances the forward and backward conditions with variance $\frac{2}{n_{in}+n_{out}}$. For ReLU, half the outputs are zeroed, so preserving variance requires $\text{Var}\left(W\right)=\frac{2}{n}$: He initialization, with the factor of two compensating for the half-zeroing.

With the right activation and the right initialization, you can train tens of layers. But not hundreds.

The degradation problem#

By 2015 the toolkit included ReLU, He initialization, and batch normalization. Then He, Zhang, Ren, and Sun went deeper: a 56-layer network against a 20-layer network on CIFAR-10, all techniques applied. The deeper network had higher error. Higher training error, not test error. Overfitting looks like low training error and high test error; this is the opposite. The optimizer was failing to find a good solution at all.

That is paradoxical. Take the trained 20-layer network, append 36 layers that each copy input to output, and you have a 56-layer network with identical training error. The deeper network should do at least as well. But SGD cannot find that solution, because learning the identity function through a stack of weights, batch normalization, and ReLU is hard. The problem is not capacity. The problem is optimization.

The skip connection#

The fix: instead of asking a block of layers to learn a mapping $H\left(x\right)$ directly, let them learn the correction $F\left(x\right)=H\left(x\right)-x$ and add the input back at the end. The block outputs $F\left(x\right)+x$. The addition is the skip connection: a wire carrying the input past the layers, no parameters, no activation, just addition.

When the optimal behavior of a block is close to the identity, a plain network must learn the identity through weights and nonlinearities, a nontrivial optimization. A residual block just needs $F\left(x\right)$ close to zero, and driving weights toward zero is what weight decay does naturally. It is much easier to learn "change nothing" when the default is already "change nothing."

The gradient argument is one line. For a block $y=F\left(x\right)+x$,

$$\frac{\mathit{\partial }y}{\mathit{\partial }x}=\frac{\mathit{\partial }F}{\mathit{\partial }x}+I$$

Even if the learned layers contribute almost nothing, the gradient includes the identity. Chain $L-m$ blocks and the product of sums expands into $2^{L-m}$ terms, one per choice of skip-or-through at each block. A plain network's gradient is a single product of Jacobians, one path, and any small factor shrinks it. A residual network's gradient is a sum of $2^{L-m}$ products, and the all-skip path contributes $I$ always, regardless of weights or depth. The gradient cannot decay exponentially with depth; it vanishes only if the learned terms conspire to cancel the identity. Sum, not product.

Pause and try: expand the gradient through two residual blocks.

Two blocks means multiplying $(\frac{\mathit{\partial }{F}_2}{\mathit{\partial }x}+I)$ by $(\frac{\mathit{\partial }{F}_1}{\mathit{\partial }x}+I)$. Four terms: both through the learned layers, $\mathit{\partial }{F}_2·\mathit{\partial }{F}_1$; first through, second skipped, $\mathit{\partial }{F}_1$; first skipped, second through, $\mathit{\partial }{F}_2$; both skipped, $I·I=I$. The gradient is the sum of all four, and the last term is the identity no matter what the weights are.

Veit, Wilber, and Belongie showed in 2016 what this path structure implies. In a 110-layer ResNet, the paths carrying most of the gradient are only about 10 to 34 layers long; the very long paths do vanish, but there are few of them. A ResNet behaves less like one very deep network and more like an ensemble of many shallow networks sharing weights. You can delete individual blocks from a trained ResNet and performance barely changes.

The proof in practice#

He et al. submitted residual networks to ImageNet in 2015. ResNet-152 achieved 5.71 percent top-5 error (10-crop, validation) as a single model, and the ensemble won the competition at 3.57 percent. Deeper consistently beat shallower, as long as the skip connections were there. A follow-up trained a 1001-layer network on CIFAR-10, keeping the shortcut a pure identity, and it converged and outperformed shallower networks. The degradation problem was gone.

The idea outlived the architecture it was born in. Every Transformer layer wraps self-attention and its feed-forward network in residual connections: the output of each sublayer is $F\left(x\right)+x$, the same formula, the same gradient logic, inside every large language model. And the heatmap experiment confirms it at small scale: the same ten-layer network with skip connections every two layers shows uniform gradient magnitude across all ten layers. No fade.

Add the input to the output. One line of arithmetic, and the gradient flows all the way through.

If you remember three things#

1. Backpropagation multiplies one Jacobian per layer, and the sigmoid derivative never exceeds $0.25$, so the gradient decays exponentially with depth: $(0.25{)}^{20}\approx {10}^{-12}$.
2. The 56-versus-20-layer experiment showed higher training error with more depth: an optimization failure, not a capacity limit, since identity layers could have matched the shallower network exactly.
3. $y=F\left(x\right)+x$ turns the gradient from one product into a sum of $2^{L-m}$ path products, and the all-skip path contributes $I$ regardless of weights or depth, so the gradient cannot decay exponentially with depth.