📐 Concept diagram

17-05 — Residual Connections

Phase: 17 — Deep Learning Architectures (Math) Subject: 17-05 Prerequisites: 16-05 (Backpropagation), 16-06 (Gradient Flow), 16-08 (Regularization — optional), 17-03/17-04 (gating concepts) Next subject: 17-06 — Attention Mechanism (General)

Learning Objectives

By the end of this subject, you will be able to:

1. Derive the residual block equation and the gradient identity that enables training of very deep networks
2. Explain why ∂y/∂x = I + ∂F(x)/∂x prevents vanishing gradients regardless of depth
3. Contrast pre-activation and post-activation residual blocks with mathematical justification
4. Prove that a residual network of depth L has at least one gradient path of length 1 (identity path)
5. Compute the number of gradient paths of each length in a residual network

Core Content

1. The Problem: Why Deep Networks Stall

From 16-06, we know that standard deep networks suffer vanishing gradients. Even with ReLU, gradients can be attenuated by weight matrices. A 100-layer network has gradient:

∂L/∂x₁ = ∏{ℓ=2}^{L} ∂h_ℓ/∂h{ℓ-1} = ∏_{ℓ=2}^{L} diag(ReLU'(z_ℓ)) · W_ℓ

If the singular values of W_ℓ are less than 1 on average, this product vanishes exponentially.

Residual connections solve this by providing an unobstructed "gradient highway" through the network — a direct identity path alongside the transformed path.

2. The Residual Block

A residual block computes:

y = F(x, {W_i}) + x

Where F is a small neural network (typically 2-3 layers) and x is the input, added back via a "skip connection" or "identity shortcut."

Shapes must match: If F changes the dimension, a learned linear projection W_s is used:

y = F(x, {W_i}) + W_s · x

⚠️ THIS IS CRITICAL — The residual connection isn't just a trick. It fundamentally changes the gradient dynamics. The derivative ∂y/∂x = I + ∂F/∂x guarantees that the gradient has a term equal to the IDENTITY MATRIX, which doesn't decay no matter how small ∂F/∂x is.

3. The Gradient Identity

Consider a network built from residual blocks. For block ℓ:

hℓ = F_ℓ(h{ℓ-1}) + h_{ℓ-1}

The Jacobian:

∂hℓ/∂h{ℓ-1} = I + ∂F_ℓ/∂h_{ℓ-1}

Now the full gradient from layer L back to layer 0:

∂L/∂h₀ = (∏{ℓ=1}^{L} (I + ∂F_ℓ/∂h{ℓ-1})) · ∂L/∂h_L

Expand the product:

∏(I + J_ℓ) = I + ΣJ_ℓ + Σ_{i ∂L/∂h₀ = Σ_{S ⊆ {1,...,L}} (∏_{ℓ∈S} J_ℓ) · ∂L/∂h_L

Where S is any subset of blocks where we take the J_ℓ path (taking I at the others).

This means there are 2^L distinct gradient paths: - 1 path of "length 0" (all identity — I) - L paths of "length 1" (one J, rest I) - C(L,2) paths of "length 2" - ... - 1 path of "length L" (all J — the vanilla deep network path)

The gradient is the SUM over all paths. Even if long paths vanish, the short paths (especially the length-0 identity path) ensure gradients flow.

5. Why This Enables 1000+ Layer Networks

In a vanilla 1000-layer network, there is exactly ONE gradient path, and every layer must successfully propagate the gradient. The probability of the entire chain working is exponentially small in L.

In a residual network, there are 2^1000 ≈ 10^301 paths. Most involve only a few J blocks and are short enough that gradients don't vanish. The network can learn even if only the short paths carry useful gradients — and gradually, as training progresses, the longer paths can be recruited.

This is the "effective depth" interpretation: early in training, the network behaves like a shallow ensemble of shorter paths; as training progresses, deeper paths become useful.

6. Pre-Activation vs Post-Activation Residual

Post-activation (original ResNet):

y = ReLU(F(x) + x)

The ReLU is applied AFTER the addition. This means the identity path also passes through ReLU, which zeros out negative values — the gradient through the identity path is gated by ReLU'.

Pre-activation (ResNet v2):

y = F(ReLU(x)) + x

The ReLU is applied BEFORE F, and the skip connection bypasses it entirely. This gives a TRUE identity path: the gradient ∂y/∂x = I + ∂F(ReLU(x))/∂x, and the I term has no ReLU' gating.

Mathematical advantage of pre-activation: The identity path is truly unobstructed. In post-activation, if the sum F(x) + x is negative, ReLU' = 0 and even the identity path is blocked.

7. The Training Dynamics Insight

A residual network with block function F(x) = W₂·σ(W₁·x) can be seen as a dynamical system:

h_{t+1} = h_t + F(h_t)

This is a forward Euler discretization of the ODE dh/dt = F(h). Very deep residual networks approximate continuous-depth models (Neural ODEs), which gives them a principled mathematical foundation.

8. Gradient Flow Quantification

For a residual block with F(x) = W₂·ReLU(W₁·x):

∂y/∂x = I + W₂·diag(ReLU'(W₁·x))·W₁

The eigenvalues of this Jacobian are 1 + λ_i, where λ_i are eigenvalues of J_F = ∂F/∂x.

If J_F is "small" (||J_F|| < 1, which is typical with proper initialization), then the eigenvalues are clustered near 1 — neither vanishing nor exploding. This is the sweet spot for gradient propagation.

Key Terms

• Residual connections

Worked Examples

Example 1: Two-Block Gradient Decomposition

Problem: A residual network with 2 blocks: h₁ = x + F₁(x), h₂ = h₁ + F₂(h₁), loss L applied at h₂. Write ∂L/∂x as a sum of path contributions.

Solution: ∂h₁/∂x = I + J₁ where J₁ = ∂F₁/∂x ∂h₂/∂h₁ = I + J₂ where J₂ = ∂F₂/∂h₁

∂L/∂x = ∂L/∂h₂ · ∂h₂/∂h₁ · ∂h₁/∂x = ∂L/∂h₂ · (I + J₂) · (I + J₁) = ∂L/∂h₂ · (I + J₁ + J₂ + J₂J₁)

Four paths: - I: gradient flows through both identity connections (∂L/∂h₂ · I) - J₁: through F₁ only (∂L/∂h₂ · J₁) - J₂: through F₂ only (∂L/∂h₂ · J₂) - J₂J₁: through both (∂L/∂h₂ · J₂J₁ — the "vanilla" deep path)

Example 2: Gradient Magnitude Bound

Problem: If ||J_ℓ|| ≤ 0.1 for all ℓ in a 100-block residual network, what is a lower bound on ||∂L/∂h₀|| relative to ||∂L/∂h₁₀₀||?

Solution: ∂L/∂h₀ = (∏(I + J_ℓ)) · ∂L/∂h₁₀₀

The identity path contributes ∂L/∂h₁₀₀ directly. So: ||∂L/∂h₀|| ≥ ||∂L/∂h₁₀₀|| − (terms from other paths)

But more precisely, if we consider the I term alone: ||∂L/∂h₀|| ≥ ||∂L/∂h₁₀₀|| (the gradient at layer 0 is at least as large as that at layer 100, via the pure identity path).

This is radically different from vanilla nets where ||∂L/∂h₀|| ≤ (0.1)^100 ||∂L/∂h₁₀₀||.

Example 3: Pre-Activation vs Post-Activation Gradient

Problem: Compute the gradient ∂y/∂x for both pre-act and post-act residual blocks. Use F(x) = W₂·σ(W₁·x) with W₁ = W₂ = I (identity) and x = [−1, 1]ᵀ. Use ReLU for σ.

Solution:

Post-activation: y = ReLU(F(x) + x) F(x) = σ(x) = ReLU([−1,1]) = [0,1] F(x) + x = [0,1] + [−1,1] = [−1,2] y = ReLU([−1,2]) = [0,2] Gradient: ∂y/∂x = diag(ReLU'(F(x)+x)) · (I + ∂F/∂x) ReLU'([−1,2]) = diag([0,1]) ∂F/∂x = diag(ReLU'([−1,1])) · I = diag([0,1]) · I = diag([0,1]) I + ∂F/∂x = diag([1,1]) + diag([0,1]) = diag([1,2]) ∂y/∂x = diag([0,1]) · diag([1,2]) = diag([0,2])

Pre-activation: y = F(ReLU(x)) + x ReLU(x) = [0,1] F(ReLU(x)) = ReLU(I·[0,1]) = [0,1] y = [0,1] + [−1,1] = [−1,2] ∂y/∂x = ∂F(ReLU(x))/∂x + I = diag(ReLU'(x)) · I · diag(ReLU'(ReLU(x))) · I + I = diag([0,1]) + I = diag([1,2])

The pre-activation gradient for the first dimension (x₁=−1): diag entry = 1 (pure identity). The post-activation: 0 (identity path blocked). Pre-activation wins for gradient flow.

Quiz

Q1: What is the Jacobian ∂y/∂x for a residual block y = F(x) + x?

A) ∂F/∂x B) I C) I + ∂F/∂x D) ∂F/∂x · I

Q2: In a residual network of depth L, how many distinct gradient paths exist from output to input?

A) L B) L² C) 2^L D) L!

Q3: What is the key advantage of pre-activation over post-activation residual blocks?

A) Fewer parameters B) The identity path bypasses the activation function entirely, so the gradient I is never gated by ReLU' C) Faster computation D) Better regularization

Q4: Why do residual connections enable 1000+ layer networks?

A) They use better activation functions B) The identity path ensures ||∂x_L/∂x₀|| ≥ 1 — gradients never vanish completely C) They reduce the number of parameters D) They eliminate the need for batch normalization

Q5: What must be done when a residual block changes the channel dimension?

A) The skip connection cannot be used B) A learned linear projection W_s · x must replace the identity skip connection C) The network must be restructured D) Zero-padding is applied to the skip

Practice Problems

Problem 1

Write the forward and backward equations for a residual block with F(x) = W₂·tanh(W₁·x). Include the skip connection.

Problem 2

How many gradient paths exist in a 3-block residual network? List them.

Problem 3

Why does pre-activation residual design give a "truer" identity path than post-activation?

Problem 4

A residual network has 50 blocks. If ||J_ℓ|| < 1 for all ℓ, what happens to the longest gradient path? What about the shortest?

Problem 5

Explain the connection between residual networks and Neural ODEs.

Summary

1. A residual block computes y = F(x) + x, creating a skip connection that adds I to the Jacobian: ∂y/∂x = I + ∂F/∂x
2. The gradient decomposes into 2^L paths — including a pure identity path (length 0) that never attenuates
3. Even if ||∂F/∂x|| ≪ 1 for all blocks, the identity path ensures gradients never vanish completely
4. Pre-activation design (y = F(σ(x)) + x) provides a cleaner identity path than post-activation (y = σ(F(x)+x))
5. Deep residual networks can be viewed as discrete ODE solvers, connecting them to continuous-depth models

Pitfalls

• Forgetting the projection shortcut when dimensions change. The identity skip connection requires matching shapes: y = F(x) + x only works when dim(F(x)) = dim(x). When F changes the channel count or spatial dimensions (e.g., stride-2 downsampling), you must add a learned linear projection W_s·x. Forgetting this causes an immediate shape mismatch error.
• Using post-activation residual blocks for very deep networks (50+ layers). Post-activation (ReLU after addition) gates the identity gradient path through ReLU', meaning negative F(x) + x values block even the skip connection's gradient. Pre-activation design (ReLU before F, skip bypasses it) provides a truly unobstructed identity path and is strongly preferred beyond ~50 layers.
• Thinking residual connections eliminate the need for careful initialization. Residuals prevent complete gradient vanishing, but if F(x) dominates the skip connection (||F(x)|| ≫ ||x||), the network behaves like a vanilla deep net with all the associated gradient problems. Proper initialization ensures the residual block starts near-identity: F(x) ≈ 0 at initialization.
• Ignoring BatchNorm placement within residual blocks. The standard pre-activation order is BN → ReLU → Conv → BN → ReLU → Conv → Add. Deviating from this (e.g., placing BN after addition, or omitting the second BN) changes the signal distribution entering each block and can degrade performance measurably.
• Adding skip connections between arbitrary layers without considering the "near-identity" prior. Residual connections work because the network can learn small corrections to an identity mapping. If the skip connection spans a large transformation (e.g., across a bottleneck with major dimension reduction), the identity interpretation breaks down and the gradient highway benefit is lost.

Next Steps

Continue to 17-06 — Attention Mechanism (General) to learn about the Query-Key-Value abstraction that powers Transformers.