📐 Concept diagram

20-02 — Gradient Clipping

Phase: 20 — Training & Fine-tuning Mathematics Subject: 20-02 Prerequisites: 20-01 (Learning Rate Schedules), 14-02 (Gradient Descent), 15-06 (Mixed-Precision Training — fp16 gradient underflow), 14-06 (Convex Sets and Functions — gradient norm), 15-04 (Backpropagation Algorithm — gradient computation) Next subject: 20-03 — Batch Size and Gradient Accumulation

Learning Objectives

By the end of this subject, you will be able to:

1. Derive the gradient norm clipping formula g_clipped = g · min(1, C/||g||) and prove it preserves gradient direction while capping magnitude
2. Analyze the three types of gradient instability that clipping addresses: exploding gradients in RNNs, mixed-precision overflow, and loss spikes in large-batch training
3. Compute the probability that an unclipped gradient exceeds threshold C given normally distributed gradient components — connecting clipping to outlier theory
4. Compare norm clipping vs value clipping (coordinate-wise) and explain when each is appropriate
5. Derive why adaptive optimizers like Adam provide implicit gradient normalization, and why explicit clipping is STILL needed in some cases

Core Content

1. The Problem: Exploding Gradients

During backpropagation, gradients can grow exponentially as they flow backward through deep networks. Consider a simple RNN unrolled for T steps with weight matrix W:

$h_t = σ(W h_{t−1} + U x_t)
$

The gradient w.r.t. loss at time T flows backward:

$∂L/∂h_{T−k} involves W^k  (approximately, ignoring activation derivatives)
$

If the largest eigenvalue of W, denoted λ_max, satisfies |λ_max| > 1, then ||∂L/∂h_{T−k}|| grows like |λ_max|^k — EXPONENTIAL growth in sequence length.

Consequence: Gradient norms can reach 10^6, 10^9, or NaN — and a single SGD step with such a gradient can irreparably damage the model. In the worst case, the optimizer follows a wild divergent path and never recovers.

This problem is most acute in: 1. RNNs/LSTMs with long sequences (classic case) 2. Transformers with very deep stacks or when training diverges (loss spikes) 3. Mixed-precision training where fp16 has limited dynamic range (max ≈ 65504) 4. Large-batch training where gradient norms tend to be larger

⚠️ THIS IS CRITICAL — Gradient clipping is a standard component in virtually all LLM training pipelines. Without it, loss spikes can destroy models mid-training.

2. Gradient Norm Clipping

The most common form: scale the gradient vector so its L2 norm does not exceed a threshold C.

$g_clipped = g · min(1, C / ||g||₂)
$

where: - g ∈ ℝ^d is the gradient vector (all parameters flattened) - ||g||₂ = √(Σ g_i²) is the L2 norm - C is the clipping threshold (hyperparameter)

How to read this: If ||g||₂ ≤ C, min(1, C/||g||) = 1, so the gradient is unchanged. If ||g||₂ > C, we scale it DOWN by factor C/||g|| < 1 so that the clipped gradient has norm exactly C.

Proof that direction is preserved:

$g_clipped = g · s      where s = min(1, C/||g||)
$

Since s > 0, g_clipped is a positive scalar multiple of g, so they point in the exact same direction. Only the step SIZE is reduced.

Proof that norm is capped:

$||g_clipped|| = ||g · s|| = s · ||g|| = min(1, C/||g||) · ||g|| = min(||g||, C)
$

So ||g_clipped|| ≤ C always. ✓

3. Per-Parameter Norm Clipping

In practice, we often clip per-parameter-group rather than over the entire flat gradient vector. For each parameter group p (e.g., all weights in a layer):

$g_p_clipped = g_p · min(1, C / max(||g_p||₂, ε))
$

This is what PyTorch's torch.nn.utils.clip_grad_norm_ does — it computes the total norm across ALL parameters by default, but this is equivalent to computing norms per-parameter-group and scaling uniformly.

Formula for total norm across parameter groups:

$||g||_total = √(Σ_p ||g_p||²)
$

Then ALL parameter groups are scaled by the same factor s = min(1, C/||g||_total). This preserves the overall direction of the gradient in parameter space.

4. Gradient Value Clipping

An alternative: clip each gradient element individually:

$g_i_clipped = clamp(g_i, −v, v)
$

where v is the per-element threshold.

Comparison with norm clipping:

When value clipping is preferred: Mixed-precision training where a few gradient elements overflow fp16's max value (65504). Value clipping to ±65504 prevents NaN while leaving most elements untouched.

5. Why Gradient Norms Spike: The Mathematics

5.1 Loss Landscape Geometry

Near sharp minima or narrow valleys, the Hessian has large eigenvalues. The gradient is:

$∇L(θ) ≈ H(θ*) · (θ − θ*)
$

Near but not AT the minimum, if H has eigenvalues λ_i ≫ 1, the gradient norm can be huge even when the loss is low. This creates a nasty feedback loop: 1. Large gradient → large step → overshoot the valley 2. Now parameters are on the OTHER side of the valley 3. Gradient is large again → another large step → oscillation or divergence

5.2 Outliers in Transformer Training

Transformer gradients have heavy-tailed distributions. Individual attention logits or FFN activations can produce gradients 100-1000× the median. The L2 norm is dominated by these outliers:

If g = [g₁, g₂, ..., g_d] and g₁ ≫ g_i for i>1,
then ||g||₂ ≈ |g₁|

A single outlier can trigger clipping of the entire gradient vector.

5.3 Connection to Mixed Precision

In fp16 (half precision): max representable ≈ 65504. A gradient of 10^5 cannot be represented — it becomes ∞ → NaN propagation. Norm clipping to C < 65504 is thus a numerical necessity for fp16 training.

6. Adaptive Optimizers and Implicit Clipping

Adam's update rule already provides a form of gradient normalization:

$Δθ = η · m̂_t / (√v̂_t + ε)
$

where m̂_t is the bias-corrected first moment and v̂_t is the bias-corrected second moment. Since each parameter's update is divided by its own gradient magnitude (√v̂_t), Adam inherently scales down parameters with consistently large gradients.

So why do we still need explicit clipping with Adam?

1. Short-term spikes: v̂_t is an exponential moving average. A single huge gradient spike hasn't yet been "averaged in" to v̂_t — the current step can still be enormous.
2. Coordinated large gradients: If many parameters have large gradients simultaneously (e.g., a loss spike), individual normalization doesn't prevent the COMBINED step from being large.
3. Numerical stability: Even if Δθ per parameter is modest, intermediate gradient values may overflow fp16 before they reach the Adam computation.

Empirical evidence: Llama, GPT-3, Chinchilla, and virtually all major LLMs use gradient clipping (typically C = 1.0) EVEN with AdamW.

7. Choosing the Clipping Threshold C

Too low: Clips almost every step → training becomes artificially slow, model may not converge because gradient direction is correct but magnitude is capped too aggressively.

Too high: Rarely activates → useless. Equivalent to no clipping.

Heuristic approaches:

1. Monitor gradient norm histogram: Pick C at roughly the 90th-95th percentile of observed norms. This clips only the most extreme steps.
2. Start high, look for spikes: If loss spikes correlate with gradient norm spikes, lower C until spikes disappear.
3. Standard values: C = 1.0 is standard for transformer LLMs. C = 0.5 is more aggressive. C = 5.0 is lenient.

Mathematical property: The optimal C depends on the Lipschitz constant of the loss. If L is L-smooth (Lipschitz gradient), gradient descent with step size η:

$L(θ_{t+1}) ≤ L(θ_t) − η||∇L||² + (L·η²/2)·||∇L||²
$

If ||∇L|| is large enough that the second term dominates, the loss INCREASES. Clipping to C ensures:

$L(θ_{t+1}) ≤ L(θ_t) − η·C² + (L·η²/2)·C²
$

This is negative (loss decreases) when η < 2/L — independent of the actual gradient magnitude, as long as C is set reasonably.

8. Gradient Clipping in LLM Training Pipelines

The standard recipe used by most LLM training code (Megatron-LM, HuggingFace Transformers, etc.):

# After loss.backward(), before optimizer.step():
total_norm = torch.nn.utils.clip_grad_norm_(
    model.parameters(), 
    max_norm=1.0,           # standard for transformers
    norm_type=2.0           # L2 norm
)
# Log total_norm for monitoring

With this setup, any step where total_norm exceeds 1.0 gets scaled down. Steps below 1.0 proceed normally. In healthy training, only ~5-10% of steps trigger clipping.

Worked Examples

Example 1: Computing Clipped Gradient

Problem: A gradient vector is g = [3.0, 4.0, 0.0, 0.0, −12.0]^T. The clipping threshold is C = 10.0. Compute g_clipped.

Solution:

$||g||₂ = √(3² + 4² + 0² + 0² + (−12)²) = √(9 + 16 + 0 + 0 + 144) = √169 = 13.0
$

Since ||g|| = 13.0 > C = 10.0, we scale:

$s = C / ||g|| = 10.0 / 13.0 ≈ 0.7692
g_clipped = s · g = 0.7692 · [3, 4, 0, 0, −12]
          = [2.308, 3.077, 0, 0, −9.231]
$

Check: ||g_clipped|| = √(2.308² + 3.077² + 0 + 0 + 9.231²) = √(5.328 + 9.468 + 85.22) = √100.016 ≈ 10.0 ✓

Example 2: Mixed-Precision Overflow Prevention

Problem: In fp16 training, the logits of a certain token reach 70000, and the corresponding gradient component in the logit layer is ∂L/∂z ≈ 35000. Can this gradient be represented in fp16? What value clipping threshold prevents overflow while leaving modest gradients (typical value ~10) unchanged?

Solution:

Can 35000 be represented? fp16 max = 65504. 35000 < 65504, so YES it can be represented. But if ∂L/∂z were 70000, it would overflow.

Value clipping threshold: To protect against overflow, use v = 65504 (fp16 max) or a safety margin like v = 60000. This clips values above this threshold to 60000 while leaving modest values (10, 100, 1000) unchanged.

But note: Norm clipping with C = 1.0 would also prevent this. If the gradient vector has norm > 1.0, ALL elements (including the 35000 outlier) get scaled down. With enough other moderate elements adding to the norm, the whole vector gets clamped. Norm clipping is a "collective" approach; value clipping is "individual."

Example 3: Exploding Gradient in Deep Linear Network

Problem: A 20-layer "linear network" (no activations) has each layer's weight initialized as W_l = 2.0·I (scalar case for simplicity). The true gradient of loss w.r.t. the last layer is ∂L/∂h_20 = 1.0. Compute the gradient w.r.t. the first layer's weight WITHOUT clipping, and determine what clipping threshold C would be needed to keep the gradient norm below 100.

Solution:

In a deep linear network f(x) = W_20 · W_19 · ... · W_1 · x, for scalar weights w_l:

$∂L/∂w₁ = (∂L/∂h_20) · ∂h_20/∂h_19 · ... · ∂h₂/∂h₁ · ∂h₁/∂w₁
       = 1.0 · w_20 · w_19 · ... · w_2 · x
       = 1.0 · (2.0)^19 · x     (since each w_i = 2.0)
       = 2^19 · x
       = 524288 · x
$

If x = 1.0, the gradient is 524,288 — way beyond any reasonable step.

Clipping threshold: To keep below 100: C = 100. Then:

$s = 100 / 524288 ≈ 1.907 × 10^{−4}
g_clipped = 524288 · 1.907e-4 = 100.0 ✓
$

This is an extreme but illustrative case — real networks have nonlinear activations that bound gradients, but deep transformers can still exhibit similar multiplicative growth.

Quiz

Q1: What does the concept of Transformers primarily refer to in this subject?

A) A visual representation of Transformers B) A computational error related to Transformers C) The definition and application of Transformers D) A historical anecdote about Transformers

Correct: C)

• If you chose A: This is incorrect. Transformers is defined as: the definition and application of transformers. The other options describe different aspects that are not the primary focus.
• If you chose B: This is incorrect. Transformers is defined as: the definition and application of transformers. The other options describe different aspects that are not the primary focus.
• If you chose C: Transformers is defined as: the definition and application of transformers. The other options describe different aspects that are not the primary focus. Correct!
• If you chose D: This is incorrect. Transformers is defined as: the definition and application of transformers. The other options describe different aspects that are not the primary focus.

Q2: What is the primary purpose of Mixed-precision training?

A) It is primarily a historical notation system B) It is used only in advanced research contexts C) It replaces all other methods in this domain D) It is used to mixed-precision training in mathematical analysis

Correct: D)

• If you chose A: This is incorrect. Mixed-precision training serves the purpose described in the correct answer. The other options misrepresent its role.
• If you chose B: This is incorrect. Mixed-precision training serves the purpose described in the correct answer. The other options misrepresent its role.
• If you chose C: This is incorrect. Mixed-precision training serves the purpose described in the correct answer. The other options misrepresent its role.
• If you chose D: Mixed-precision training serves the purpose described in the correct answer. The other options misrepresent its role. Correct!

Q3: Which statement about Large-batch training is TRUE?

A) Large-batch training is mentioned only as a historical footnote B) Large-batch training is an advanced topic beyond this subject's scope C) Large-batch training is a fundamental concept covered in this subject D) Large-batch training is not related to this subject

Correct: C)

• If you chose A: This is incorrect. Large-batch training is a fundamental concept covered in this subject. This subject covers Large-batch training as part of its core content.
• If you chose B: This is incorrect. Large-batch training is a fundamental concept covered in this subject. This subject covers Large-batch training as part of its core content.
• If you chose C: Large-batch training is a fundamental concept covered in this subject. This subject covers Large-batch training as part of its core content. Correct!
• If you chose D: This is incorrect. Large-batch training is a fundamental concept covered in this subject. This subject covers Large-batch training as part of its core content.

Q4: Based on the worked examples in this subject, what is the correct result?

A) A different result from a common mistake B) ||x|| ≤ C}: C) An unrelated numerical value D) The inverse of the correct answer

Correct: B)

• If you chose A: This is incorrect. The worked examples show that the result is ||x|| ≤ C}:. The other options represent common errors.
• If you chose B: The worked examples show that the result is ||x|| ≤ C}:. The other options represent common errors. Correct!
• If you chose C: This is incorrect. The worked examples show that the result is ||x|| ≤ C}:. The other options represent common errors.
• If you chose D: This is incorrect. The worked examples show that the result is ||x|| ≤ C}:. The other options represent common errors.

Q5: How are Large-batch training and Gradient norm clipping related?

A) Large-batch training is a special case of Gradient norm clipping B) Large-batch training and Gradient norm clipping are completely unrelated topics C) Large-batch training is the inverse of Gradient norm clipping D) Large-batch training and Gradient norm clipping are closely related concepts

Correct: D)

• If you chose A: This is incorrect. Both Large-batch training and Gradient norm clipping are covered in this subject as interconnected topics.
• If you chose B: This is incorrect. Both Large-batch training and Gradient norm clipping are covered in this subject as interconnected topics.
• If you chose C: This is incorrect. Both Large-batch training and Gradient norm clipping are covered in this subject as interconnected topics.
• If you chose D: Both Large-batch training and Gradient norm clipping are covered in this subject as interconnected topics. Correct!

Q6: What is a common pitfall when working with Clipping is essential?

A) The main error with Clipping is essential is using it when it is not needed B) A common mistake is confusing Clipping is essential with a similar concept C) Clipping is essential is always computed the same way in all contexts D) Clipping is essential has no common misconceptions

Correct: B)

• If you chose A: This is incorrect. Students often confuse Clipping is essential with similar-sounding or related concepts. Pay attention to the precise definitions.
• If you chose B: Students often confuse Clipping is essential with similar-sounding or related concepts. Pay attention to the precise definitions. Correct!
• If you chose C: This is incorrect. Students often confuse Clipping is essential with similar-sounding or related concepts. Pay attention to the precise definitions.
• If you chose D: This is incorrect. Students often confuse Clipping is essential with similar-sounding or related concepts. Pay attention to the precise definitions.

Q7: When should you apply Norm clipping preserves gradient direction?

A) Apply Norm clipping preserves gradient direction to solve problems in this subject's domain B) Norm clipping preserves gradient direction is not practically useful C) Avoid Norm clipping preserves gradient direction unless explicitly instructed D) Use Norm clipping preserves gradient direction only in pure mathematics contexts

Correct: A)

• If you chose A: Norm clipping preserves gradient direction is a practical tool used throughout this subject to solve relevant problems. Correct!
• If you chose B: This is incorrect. Norm clipping preserves gradient direction is a practical tool used throughout this subject to solve relevant problems.
• If you chose C: This is incorrect. Norm clipping preserves gradient direction is a practical tool used throughout this subject to solve relevant problems.
• If you chose D: This is incorrect. Norm clipping preserves gradient direction is a practical tool used throughout this subject to solve relevant problems.

Practice Problems

Problem 1

g = [1, 2, 2, 4, 4, 8], C = 5.0. Compute g_clipped and verify the norm equals C.

$||g|| = √(1 + 4 + 4 + 16 + 16 + 64) = √105 ≈ 10.247
s = 5.0 / 10.247 ≈ 0.488
g_clipped = 0.488 · [1,2,2,4,4,8] = [0.488, 0.976, 0.976, 1.952, 1.952, 3.904]
||g_clipped|| ≈ √(0.238 + 0.952 + 0.952 + 3.809 + 3.809 + 15.241) = √25.001 ≈ 5.0 ✓
$

Problem 2

Prove that norm clipping with threshold C followed by an SGD update of step size η is equivalent to projecting the gradient onto the ball of radius C, then taking the step. What optimization problem does this correspond to?

$proj_{B_C}(g) = argmin_{x: ||x||≤C} ||x − g||²
$

$min_θ L(θ)   subject to   each update Δθ has norm ≤ η·C
$

Problem 3

In mixed-precision training, gradients are stored in fp16 (max 65504) while optimizer states are in fp32. A training step produces a gradient vector of norm 50,000. The clipping threshold is C = 1.0. Answer: (a) Will the unclipped gradient fit in fp16? (b) What norm will the clipped gradient have? (c) If we ALSO apply value clipping at v = 65504, is it redundant with norm clipping here?

Problem 4

A loss spike occurs: the gradient norm jumps from a typical 0.3 to 25.0 in one step. With clipping threshold C = 1.0, what factor is the gradient scaled by? If the "true" gradient direction was correct but the magnitude was inflated by a numerical issue, does clipping help or hurt?

$s = 1.0 / 25.0 = 0.04
$

Problem 5

Consider g ~ N(0, σ²I_d) (gradients independently normally distributed). For d = 1000 and σ² = 1/1000 (so E[||g||²] = 1), use the chi-squared distribution to find the probability that ||g|| > 2.0. What does this say about how often clipping with C = 2.0 would activate under normality assumptions?

$||g||² ~ (σ²) · χ²_d = (1/1000) · χ²_1000
$

Summary

1. Gradient norm clipping scales the entire gradient vector by min(1, C/||g||) when ||g|| exceeds C — preserving direction while capping magnitude
2. Clipping is essential for RNNs (exploding gradients), mixed-precision training (fp16 overflow), and large-batch transformer training (loss spikes)
3. Norm clipping preserves gradient direction while value clipping handles per-element overflow — they serve complementary roles
4. Adam's adaptive normalization is NOT a substitute for explicit clipping — short-term spikes bypass the EMA smoothing, and coordinated large gradients still cause instability
5. Standard LLM recipe: norm clipping with C = 1.0, typically activated on <10% of steps in healthy training

Pitfalls

• Relying on Adam's implicit normalization as a substitute for explicit gradient clipping. Adam divides each parameter's update by √v̂_t, which normalizes based on HISTORICAL gradient magnitude. A single gradient spike contributes only ~0.1% to v̂_t (since 1−β₂ ≈ 0.001), so the current step can still be enormous. Virtually all major LLMs (GPT-3, Llama, Chinchilla) use explicit norm clipping in addition to AdamW — they address different problems.
• Setting the clipping threshold too high (e.g., C = 100). If C ≫ typical ||g||, clipping never activates and provides no protection. The standard value C = 1.0 works because it clips only the heaviest tail of the gradient distribution (~5-10% of steps). Monitor gradient norm histograms to verify that your C clips outliers without affecting normal steps.
• Using value clipping (clamp to ±v) when norm clipping is needed. Value clipping modifies individual gradient components independently, changing the gradient's DIRECTION. Norm clipping scales everything uniformly, preserving the direction while capping magnitude. For training stability (exploding gradients, loss spikes), norm clipping is almost always the right choice. Value clipping is for fp16 overflow prevention specifically.
• Assuming fp16 overflow won't happen because "gradients are small." Even when the L2 norm appears moderate, individual gradient components can exceed fp16's max (65504). This is especially common in the logit layer where per-token gradients spike on rare tokens. A single overflow produces NaN, which propagates through all parameters after one optimizer step. Always clip before converting gradients to fp16.
• Not monitoring gradient norm histograms during training. Without monitoring, you can't tell whether clipping is too aggressive (clipping every step, slowing training) or too lenient (loss spikes slip through). A healthy training run shows a gradient norm distribution with a long right tail, with clipping activating at roughly the 90th-95th percentile.

Key Terms

Next Steps

Continue to 20-03 — Batch Size and Gradient Accumulation to understand how large effective batch sizes are achieved and the mathematical relationship between batch size, gradient noise, and training dynamics.