📐 Concept diagram

16-06 — Gradient Flow in Deep Networks

Phase: 16 — Neural Network Mathematics Subject: 16-06 Prerequisites: 16-05 (Backpropagation), 16-02 (Activation Functions), Phase 9 (matrix norms) Next subject: 16-07 — Weight Initialization

Learning Objectives

By the end of this subject, you will be able to:

1. Prove mathematically why sigmoid and tanh activations cause vanishing gradients in deep networks
2. Explain how ReLU mitigates vanishing gradients and quantify the improvement
3. Derive the conditions for exploding gradients and explain gradient clipping
4. Analyze gradient magnitude through a network using the product-of-Jacobians decomposition
5. Connect initialization strategies to stable gradient flow at initialization time

Core Content

1. The Gradient as a Product of Jacobians

Recall from 16-05 that the error signal propagates backward:

δ^(ℓ) = ((W^(ℓ+1))ᵀ δ^(ℓ+1)) ⊙ f'_ℓ(z^(ℓ))

Unrolling this from the output back to layer ℓ:

δ^(ℓ) = (∏_{k=ℓ+1}^{L} diag(f'_k(z^(k))) (W^(k))ᵀ) δ^(L)

This is a long product of matrices. The norm of this product determines whether gradients survive or die.

⚠️ THIS IS CRITICAL — Gradient flow is determined by the product of Jacobians along the backward path. If the eigenvalues of these Jacobians are consistently < 1, gradients VANISH. If they're > 1, gradients EXPLODE. Stable training requires eigenvalues ≈ 1 on average.

2. Vanishing Gradients: The Sigmoid/Tanh Problem

For a deep network using sigmoid activations, let's analyze the gradient magnitude for early layers.

Consider the simplified case where all layers have the same width and weights are initialized with small values (a common early practice). The backward recurrence is:

δ^(ℓ) = Wᵀ δ^(ℓ+1) ⊙ σ'(z^(ℓ))

Taking norms and assuming independence:

||δ^(ℓ)|| ≈ ||W|| · σ'_max · ||δ^(ℓ+1)||

Where σ'_max = max_z σ'(z) = 0.25.

Even if ||W|| = 1, each layer multiplies the gradient by at most 0.25. After L layers:

||δ^(0)|| ≈ (0.25)^L · ||δ^(L)||

For a 10-layer network: (0.25)¹⁰ ≈ 9.5 × 10⁻⁷ — the gradient reaching the first layer is a MILLIONTH of the output gradient. The early layers effectively receive zero learning signal.

Why tanh helps only slightly: tanh'(z) ≤ 1 (max at z=0), but saturates quickly. For random inputs z ∼ N(0, 1), the expected tanh'(z) is about 0.3-0.5 — still less than 1. After many layers, gradients still vanish, just more slowly than with sigmoid.

3. ReLU to the Rescue

ReLU has derivative 1 for ALL positive inputs:

ReLU'(z) = 1 for z > 0

If the network is initialized so that roughly half the neurons are active (z > 0), the expected Jacobian norm product doesn't decay:

δ^(ℓ) = Wᵀ δ^(ℓ+1) ⊙ ReLU'(z^(ℓ))

The ReLU'(z) factor is 1 for active neurons and 0 for inactive ones. For the active paths, gradients propagate with NO multiplicative attenuation from the activation.

The catch — dying ReLUs: If initialization or learning pushes too many neurons permanently into the negative regime, those path gradients are permanently 0. The effective depth is reduced, but with proper initialization (16-07), this is manageable.

Quantitative comparison: For a 100-layer network: - Sigmoid: gradient to layer 1 ≈ (0.25)¹⁰⁰ ≈ 6 × 10⁻⁶¹ — completely gone - ReLU (50% active): gradient to layer 1 passes through ∼50 active × 1.0 multiplications — full magnitude preserved along active paths!

4. Exploding Gradients

The opposite problem: if weight matrices have large singular values, gradients GROW exponentially:

||δ^(ℓ)|| ≈ (σ_max(W))^L · ||δ^(L)||

If σ_max(W) > 1, gradients explode. This is especially problematic in RNNs (unrolled through time) and very deep networks.

Symptoms of exploding gradients: - Loss suddenly jumps to NaN - Weight updates become enormous - Training diverges catastrophically

Solution — Gradient Clipping:

g ← g · min(1, C / ||g||)

If the gradient norm ||g|| exceeds threshold C, scale it down to norm C while preserving direction. This prevents any single update from being destructively large.

Alternative — value clipping:

gᵢ ← clip(gᵢ, −C, C)

Clips each gradient component individually to [−C, C].

5. The Gradient Norm Across Layers

A useful diagnostic: plot ||∂L/∂W^(ℓ)|| for each layer ℓ during training.

Expected healthy behavior: Gradient norms are roughly similar across layers (within an order of magnitude).

Vanishing gradient signature: Early layers have dramatically smaller gradient norms than later layers — sometimes 10⁻¹⁰× smaller.

Exploding gradient signature: Early layers have dramatically larger gradient norms than later layers.

6. Mathematical Analysis of Gradient Flow at Initialization

At initialization (before training), we can analyze gradient flow analytically. Consider a linear network with orthogonal weight matrices (WᵀW = I) of equal dimension:

a^(ℓ) = W^(ℓ) a^(ℓ−1)

The gradient: ∂L/∂a^(0) = (W^(L))ᵀ (W^(L−1))ᵀ ... (W^(1))ᵀ ∂L/∂a^(L)

Since each W^(ℓ) is orthogonal, ||∂L/∂a^(0)|| = ||∂L/∂a^(L)|| — perfect gradient preservation!

With non-linear activations: Even with orthogonal weights, activations introduce scaling. The Jacobian of the activation f at layer ℓ has eigenvalues f'(zᵢ^(ℓ)). The product:

||∂L/∂a^(0)|| = (∏{ℓ=1}^{L} ∏{i} |f'(zᵢ^(ℓ))|)^{1/d} · ||∂L/∂a^(L)||

If f = ReLU and half the neurons are active: each active path contributes factor 1. The gradient magnitude at the input is approximately (1/2)^{L/2} · ∂L/∂a^(L) — decaying, but MUCH more slowly than with sigmoid.

7. Residual Connections and Gradient Highways

Residual connections (formalized in Phase 17) provide a direct gradient path:

a^(ℓ+1) = F(a^(ℓ)) + a^(ℓ)

The backward gradient:

∂L/∂a^(ℓ) = ∂L/∂a^(ℓ+1) · (∂F/∂a^(ℓ) + I) = ∂L/∂a^(ℓ+1) + ∂L/∂a^(ℓ+1)·∂F/∂a^(ℓ)

The identity term (I) provides a "gradient highway" — even if ∂F/∂a^(ℓ) has small eigenvalues, the +I term ensures at least that portion of the gradient reaches earlier layers unchanged. This is why ResNets can be 1000+ layers deep while plain networks fail beyond ~20 layers.

Key Terms

• Exploding gradients
• Gradient norm after L layers
• Monitoring gradient norms per layer
• ReLU mitigates vanishing gradients
• Residual connections

Worked Examples

Example 1: Computing Gradient Attenuation

Problem: A 5-layer network with sigmoid activations has all pre-activations z ∼ N(0, 1) i.i.d. Assume orthogonal W matrices with ||W|| = 1. What fraction of the output gradient reaches the input?

Solution:

For z ∼ N(0,1), we need E[σ'(z)]. σ'(z) = σ(z)(1−σ(z)).

Using numerical approximation: for z ∼ N(0,1), the expected value of σ'(z) is approximately 0.207 (can be computed via integration or sampling).

Per layer: gradient multiplied by ∼0.207 on average. After 5 layers: 0.207⁵ ≈ 0.00038.

Only 0.038% of the gradient survives. The first layer learns 2600× slower than the last.

With ReLU instead (half active): per-layer factor is 0.5 (only active paths survive). After 5 layers with ReLU: 0.5⁵ = 0.03125. About 3% survives — 82× more than with sigmoid.

Example 2: Exploding Gradients in an RNN

Problem: An RNN with scalar hidden state h_t = w·h_{t−1} + ... is unrolled for T = 100 steps. The recurrent weight w = 1.1. The loss depends on the final state h_T. What is ∂L/∂h₀?

Solution:

∂L/∂h₀ = ∂L/∂h_T · ∂h_T/∂h₀

∂h_T/∂h₀ = ∏{t=1}^{T} ∂h_t/∂h{t−1} = ∏_{t=1}^{T} w = w^T

With w = 1.1, T = 100: 1.1¹⁰⁰ ≈ 13,780.

The gradient is amplified by a factor of nearly 14,000! This causes wildly unstable training. Gradient clipping would cap this at C/||g||.

If w = 0.9 instead: 0.9¹⁰⁰ ≈ 2.66 × 10⁻⁵ — nearly zero. The RNN forgets the distant past.

This is the fundamental challenge of training RNNs: the weight magnitude must be very close to 1 for long-range dependencies, which is why LSTMs/GRUs (Phase 17) use gating mechanisms to learn when to remember/forget rather than relying on a fixed scalar multiplier.

Example 3: Gradient Clipping Threshold Selection

Problem: During training, you observe that gradient norms fluctuate between 0.1 and 50. You want to clip to prevent the largest updates from destabilizing training. The average healthy gradient norm is around 1. What threshold C should you choose, and what would be the effective update step for a gradient with ||g|| = 50 when C = 5?

Solution:

Choose C around the 90th-95th percentile of gradient norms, not the average. If norms range from 0.1 to 50 with median near 1, try C = 5.

For ||g|| = 50 with C = 5: g_clipped = g · (5/50) = 0.1 · g

The update direction is preserved, but the step size is reduced by 10×. This prevents a single outlier batch from destroying progress while still allowing learning from it.

Typical values in practice: C = 1.0, C = 5.0, or C = 10.0, tuned based on monitoring.

Practice Problems

(Answers are below. Try each problem before checking.)

Problem 1: For a 20-layer ReLU network where exactly 40% of neurons are active at each layer (all weights orthogonal, ||W||=1), compute the expected fraction of gradient paths that survive from the output to the first layer.

Problem 2: A deep linear network has weight matrices with singular values all equal to σ. Derive the condition on σ for neither vanishing nor exploding gradients.

Problem 3: In a network with tanh activation, the pre-activations are distributed as z ∼ Uniform(−2, 2) at each layer. Estimate the per-layer gradient attenuation factor E[tanh'(z)].

Problem 4: Gradient clipping with threshold C scales g to have norm C when ||g|| > C. Prove that this operation is a projection onto the ℓ₂ ball of radius C.

Problem 5: Show that in a residual network with update a^(ℓ+1) = F_ℓ(a^(ℓ)) + a^(ℓ), the gradient ∂L/∂a^(ℓ) contains an additive term ∂L/∂a^(ℓ+1) that does NOT pass through the Jacobian of F_ℓ. Explain why this prevents vanishing gradients even if F_ℓ has very small Jacobian.

Summary

1. Gradient norm after L layers = product of Jacobian norms. Sigmoid (max derivative 0.25) causes exponential decay; tanh is slightly better but still problematic beyond ~10 layers.
2. ReLU mitigates vanishing gradients because its derivative is exactly 1 for active neurons, but "dying ReLUs" can block paths entirely.
3. Exploding gradients occur when ||W|| > 1; solved by gradient clipping: scale down any gradient exceeding threshold C.
4. Residual connections add an identity term to the gradient path, ensuring ∂L/∂a^(ℓ) receives ∂L/∂a^(ℓ+1) directly — a gradient highway that prevents vanishing even in very deep networks.
5. Monitoring gradient norms per layer is the primary diagnostic: healthy networks show similar magnitudes across all layers.

Pitfalls

• Confusing dying ReLUs with vanishing gradients. A dying ReLU (permanent zero output) is a forward-pass problem caused by poor initialization or large negative biases — the neuron stops contributing entirely. Vanishing gradients is a backward-pass attenuation problem where the signal shrinks exponentially with depth. They have different causes and different fixes.
• Assuming residual connections eliminate all gradient issues. Residuals provide an additive identity path that prevents gradient vanishing, but they don't fix exploding gradients. If weight matrix norms exceed 1, gradient clipping is still essential even in residual networks.
• Setting gradient clipping threshold C too low. If C is below the typical healthy gradient norm, every step is clipped, crippling learning. Monitor gradient norms during the first few hundred training steps and set C around the 90th-95th percentile — not the average.
• Overestimating tanh's help with vanishing gradients. tanh'(z) ≤ 1 (max at z=0) with expected value ~0.3–0.5 for typical inputs. This is better than sigmoid's 0.25, but still causes exponential decay over many layers. A 50-layer tanh network still vanishes; ReLU or residuals are needed for depth.
• Analyzing gradient flow only at initialization. Gradient flow evolves during training as weights change. A network stable at initialization can develop vanishing or exploding gradients mid-training. Monitor per-layer gradient norms throughout training, not just at step 0.

Quiz

Q1: In a 50-layer sigmoid network with random weight matrices (assume ||W|| ≈ 1), what is the approximate gradient magnitude at layer 1 relative to layer 50?

A) About the same B) About 0.25⁴⁹ ≈ 0 C) About 4⁴⁹ (exploded) D) It depends only on the loss function

Q2: What is the primary advantage of ReLU over sigmoid for gradient flow?

A) ReLU is faster to compute B) ReLU's derivative is exactly 1 for positive inputs, eliminating multiplicative attenuation C) ReLU guarantees all gradients are positive D) ReLU has a smaller output range

Q3: Gradient clipping with threshold C:

A) Changes the direction of the gradient B) Scales the gradient to have norm C if ||g|| > C C) Sets all gradients to exactly C D) Only clips positive gradients

Q4: Why do residual connections (skip connections) help with vanishing gradients?

A) They add more parameters to learn B) They provide an additive identity path in the gradient, bypassing the Jacobian of the residual function C) They make the network deeper D) They replace ReLU with a better activation

Q5: An RNN uses the recurrence h_t = σ(w·h_{t−1} + U·x_t). For the gradient ∂L/∂h₀ to neither vanish nor explode over T=1000 steps, what must the recurrent weight w approximately satisfy?

A) w ≈ 0 B) w ≈ 1 C) w > 4 to overcome sigmoid saturation D) Any value works as long as the network is trained long enough

Next Steps

Move on to 16-07 — Weight Initialization to learn how proper initialization (Xavier/Glorot, He/Kaiming) sets up the initial gradient flow to be stable from the very first training step.