📐 Concept diagram

25-04 — Double Descent

Phase: 25 — Frontiers & Active Research Areas Subject: 25-04 Prerequisites: Phase 14 (Optimization), Phase 12 (Information Theory) Next subject: 25-05 — Mode Connectivity and Loss Landscapes

Learning Objectives

By the end of this subject, you will be able to:

1. State the classical bias-variance tradeoff and explain why double descent contradicts it
2. Distinguish model-wise, sample-wise, and epoch-wise double descent
3. Derive conditions under which double descent occurs in linear regression (the textbook case)
4. Analyze the interpolation threshold and its role in the double descent curve
5. Evaluate explanations for double descent including implicit regularization and benign overfitting

Core Content

The Classical Bias-Variance Tradeoff

Classical statistical learning theory posits the U-shaped risk curve: as model complexity increases, test error first decreases (reducing bias), then increases (increasing variance). The optimal model complexity lies at the bottom of the U.

$$\mathbb{E}[\text{Test Error}] = \underbrace{\text{Bias}^2}{\text{decreases with complexity}} + \underbrace{\text{Variance}}{\text{increases with complexity}} + \underbrace{\text{Irreducible Error}}_{\text{Bayes error}}$$

This predicts monotonic behavior: once you pass the optimal point, larger models should get worse.

⚠️ CRITICAL: The classical U-curve was taught as gospel for decades. Double descent says this picture is incomplete for modern overparameterized models.

What Is Double Descent?

Double descent (Belkin et al., 2019) describes a risk curve with two descents as model complexity increases:

1. First descent (classical regime): Test error decreases as model complexity increases. This is the left side of the U-curve. Minimum reached near the point where the model has just enough capacity.

2. Second descent (modern regime): Test error increases near the interpolation threshold (where the model first achieves zero training error), then decreases again as complexity increases further into the overparameterized regime.

The full curve: 📉 → 📈 → 📉 (decrease, increase, decrease).

Three Varieties of Double Descent

Double descent has been observed in three dimensions:

1. Model-wise double descent (original): Vary model size (number of parameters). Test error shows the double-descent shape as a function of parameter count.

2. Sample-wise double descent: Vary dataset size $n$ for a fixed model. For small $n$ (underparameterized), test error is high. For $n$ near the parameter count (interpolation threshold), test error peaks. For $n \gg$ parameter count, test error decreases again.

3. Epoch-wise double descent: For a fixed large model, test error can show double descent over training epochs. After an initial decrease, error rises (overfitting), then decreases again as training continues.

⚠️ The epoch-wise variant connects directly to grokking (25-03) — the final descent after overfitting is a related phenomenon.

The Interpolation Threshold

The interpolation threshold is the point where the model first achieves (near) zero training error. In the model-wise case, this occurs when the number of parameters equals the number of training examples (roughly). At this point:

$$N_{\text{params}} \approx N_{\text{train}}$$

The model has just enough capacity to memorize the training set perfectly. This is the peak of test error — counterintuitively, adding more parameters after this point reduces test error.

Mathematical Analysis: Linear Regression

The cleanest demonstration of double descent is in minimum-norm least squares (linear regression in the overparameterized regime).

Given data matrix $X \in \mathbb{R}^{n \times d}$ (n samples, d features) and targets $\mathbf{y} \in \mathbb{R}^n$, consider the minimum-norm solution:

$$\hat{\boldsymbol{\beta}} = \arg\min_{\boldsymbol{\beta}} |\boldsymbol{\beta}|_2 \quad \text{subject to} \quad X\boldsymbol{\beta} = \mathbf{y}$$

When $d \leq n$ (underparameterized), the solution is the standard OLS estimator $\hat{\boldsymbol{\beta}} = (X^T X)^{-1} X^T \mathbf{y}$ (assuming full rank). When $d > n$ (overparameterized), there are infinitely many solutions with zero training error; the minimum-norm solution is:

$$\hat{\boldsymbol{\beta}} = X^T (X X^T)^{-1} \mathbf{y}$$

The test risk (expected squared error on new data) can be computed analytically for Gaussian data. Let $\mathbf{x}{\text{test}} \sim \mathcal{N}(0, I_d)$ and $y{\text{test}} = \mathbf{x}_{\text{test}}^T \boldsymbol{\beta}^* + \varepsilon$ with $\varepsilon \sim \mathcal{N}(0, \sigma^2)$. Then:

Underparameterized regime ($d < n$): $$R(d) = \sigma^2 \cdot \frac{d}{n-d-1}$$

This decreases as $d$ increases? No — it increases as $d \to n$ (diverges at the interpolation threshold).

At the threshold ($d = n$): $R \to \infty$ — the variance blows up.

Overparameterized regime ($d > n$): $$R(d) = (1 - \frac{n}{d})|\boldsymbol{\beta}^*|_2^2 + \sigma^2 \cdot \frac{n}{d-n-1}$$

As $d \to \infty$, the risk approaches $|\boldsymbol{\beta}^|_2^2 + 0 = |\boldsymbol{\beta}^|_2^2$. This is the second descent — adding more features (beyond the interpolation threshold) reduces test error.

Key insight: The minimum-norm requirement acts as an implicit regularizer. In the overparameterized regime, among all interpolating solutions, gradient descent (and the pseudo-inverse) picks the one with minimum $\ell_2$ norm. This minimum-norm solution generalizes better than the "average" interpolating solution.

Explanations for Double Descent

1. Implicit regularization of optimization: Gradient descent on overparameterized models implicitly converges to the minimum-norm (or max-margin) solution. This built-in regularization prevents the model from picking a "bad" interpolating solution.

2. Benign overfitting: In high dimensions, it's possible to fit noise perfectly (overfit) while still generalizing well — a phenomenon called benign overfitting. Conditions: the signal is aligned with high-variance directions of the data, and the noise is isotropic.

3. Model capacity and effective complexity: The "effective complexity" (what the model actually uses) may be much lower than the parameter count. Overparameterization provides a rich space of solutions, and optimization dynamics select the simplest one.

4. Double descent as a phase transition: The interpolation threshold is a phase transition in the random matrix theory sense. The risk exhibits non-monotonic behavior with a singularity (divergence) at the threshold for noise-free problems, which is smoothed by regularization.

The Role of Label Noise

Double descent is most pronounced with label noise. With clean data, the peak at the interpolation threshold is smaller or absent. With noisy labels, the model must decide whether to fit the noise (memorize), causing a spike in test error at the threshold.

For a dataset with noise variance $\sigma^2_\varepsilon$, the peak height scales with $\sigma^2_\varepsilon$ — noisier data → higher peak.

Key Terms

• Double descent
• Implicit regularization
• Label noise
• The epoch-wise variant connects directly to grokking
• U-shaped risk curve

Worked Examples

Example 1: Analytical Risk for Simple Linear Case

Problem: Consider $d=1$ feature, $\beta^* = 3$, $\sigma^2 = 0.5$, and $n=5$ training points. Compute the expected test risk (MSE) for the OLS estimator. Then consider an overparameterized case with $d=10$ features (only the first is informative) and $n=5$ samples — what happens?

Solution: Underparameterized ($d=1$, $n=5$): $$R = \sigma^2 \cdot \frac{d}{n-d-1} = 0.5 \cdot \frac{1}{5-1-1} = 0.5 \cdot \frac{1}{3} \approx 0.167$$

At threshold ($d=5=n$): The formula diverges — $R \to \infty$. In practice, the minimum-norm solution still exists but has enormous variance.

Overparameterized ($d=10$, $n=5$): $$R = (1 - \frac{5}{10}) \cdot 3^2 + 0.5 \cdot \frac{5}{10-5-1} = 0.5 \cdot 9 + 0.5 \cdot \frac{5}{4} = 4.5 + 0.625 = 5.125$$

As $d \to \infty$: $R \to |\beta^*|_2^2 = 9$. So very overparameterized models ($d \gg n$) have risk approaching the signal energy, which for uninformative features can be high.

Example 2: Identifying Double Descent in Experiments

Problem: You train ResNets of varying widths on CIFAR-10. The test errors are: - Width 4 (0.03M params): 45% - Width 8 (0.12M): 28% - Width 16 (0.47M): 12% - Width 32 (1.8M): 8% - Width 48 (4.0M): 15% ← peak! - Width 56 (5.5M): 9% - Width 64 (7.0M): 6% - Width 128 (28M): 5%

Is this double descent? Where is the interpolation threshold?

Solution: Yes, this is classic model-wise double descent. The curve: 45% → 28% → 12% → 8% (first descent), then 15% (peak at width 48), then 9% → 6% → 5% (second descent). The interpolation threshold is around width 48 (~4M parameters), where the model first achieves near-zero training error. After this point, larger models generalize better.

Example 3: Epoch-wise Double Descent and Grokking

Problem: Contrast epoch-wise double descent with grokking (25-03). A transformer on language modeling shows: - Epochs 1-5: test perplexity drops from 500 → 50 - Epochs 5-20: test perplexity rises to 80 - Epochs 20-100: test perplexity drops to 25

Is this epoch-wise double descent, grokking, or both?

Solution: This looks more like epoch-wise double descent than grokking, because: 1. Test perplexity never stays at "chance level" — it drops to 50 before rising 2. The 5→20 epoch degradation is classical overfitting (not a memorization plateau) 3. The final descent from 80→25 is the "second descent" of epoch-wise double descent

Grokking would require test accuracy to stay at chance (perplexity near vocabulary size, e.g., 10K+) during the initial memorization phase, then jump sharply. This example shows a more gradual, multi-phase pattern.

Practice Problems

Problem 1: For linear regression with Gaussian features, compute the risk $R(d)$ as $d \to n$ from below and above. Why does it diverge at exactly $d = n$?

Problem 2: Explain why double descent does NOT contradict the bias-variance decomposition, even though it contradicts the classical U-shaped risk curve.

Problem 3: You train random Fourier features on a regression task with varying numbers of features $d$, keeping $n=1000$ fixed. At what value of $d$ would you expect to see the interpolation threshold? What would the double descent curve look like qualitatively?

Problem 4: How does label noise affect the double descent curve? If a dataset has 20% label noise, how would the peak at the interpolation threshold compare to a clean dataset?

Problem 5: A researcher claims: "Double descent proves that larger models are always better." Critique this statement using the concepts from this subject.

Summary

• Double descent contradicts the classical U-shaped risk curve: test error decreases, increases near the interpolation threshold, then decreases again
• Three varieties: model-wise (vary parameters), sample-wise (vary dataset size), epoch-wise (vary training time)
• The interpolation threshold (where training error first hits zero) is the peak of test error — the "danger zone"
• Implicit regularization (minimum-norm solutions, gradient descent dynamics) causes the second descent by selecting well-generalizing solutions among all interpolating ones
• The phenomenon is cleanest in linear regression with minimum-norm least squares, where the risk formula shows explicit divergence at the threshold and decay afterward
• Label noise amplifies the peak; without noise, the effect can be subtle or absent

Quiz

Question 1: What is the "interpolation threshold" in double descent?

A. The point where test error is minimized B. The point where the model first achieves zero training error C. The point where the model has the fewest parameters D. The point where bias and variance are equal

Correct Answer: B

Question 2: In model-wise double descent, what happens to test error when you INCREASE model size past the interpolation threshold?

A. Test error increases monotonically (classical overfitting) B. Test error decreases (second descent) C. Test error stays constant D. Test error oscillates randomly

Correct Answer: B

Question 3: For minimum-norm least squares with $d$ features and $n$ training samples, the risk diverges when:

A. $d \ll n$ B. $d = n$ (exactly at the interpolation threshold) C. $d \gg n$ D. $d = 0$

Correct Answer: B

Question 4: Which form of double descent is most closely related to grokking?

A. Model-wise double descent B. Sample-wise double descent C. Epoch-wise double descent D. Architecture-wise double descent

Correct Answer: C

Question 5: Why does the minimum-norm interpolating solution generalize better than an arbitrary interpolating solution?

A. It has the smallest training error B. Implicit regularization: the minimum-norm constraint acts like $\ell_2$ regularization, favoring simpler functions C. It uses more parameters than other solutions D. It always achieves zero test error

Correct Answer: B

Question 6: How does label noise affect the double descent peak?

A. It eliminates the peak entirely B. It amplifies the peak — noisier data makes the spike taller C. It shifts the peak to a different parameter count D. It has no effect

Correct Answer: B

Pitfalls

1. Thinking double descent replaces the bias-variance decomposition: It doesn't — the decomposition remains a mathematical identity. Double descent revises our model of how bias/variance scale with complexity.
2. Assuming all large models benefit from double descent: Without proper regularization (implicit or explicit), very large models can overfit badly. The optimization algorithm matters.
3. Ignoring the interpolation threshold as a danger zone: For a given dataset, there may be model sizes that are WORSE than both smaller and larger alternatives.
4. Generalizing from clean data experiments: Double descent is most dramatic with label noise. On clean data, the peak may be subtle.

Pitfalls

1. Confusing double descent with "bigger models are always better": Double descent shows a non-monotonic risk curve — there's a dangerous regime near the interpolation threshold where models perform WORSE than both smaller and larger alternatives. The correct takeaway isn't "always go bigger," but "avoid the critical regime where $N_{\text{params}} \approx N_{\text{train}}$." Practical model selection should consider this peak.

2. Ignoring the role of the optimization algorithm: The second descent depends critically on the optimizer finding the minimum-norm (or max-margin) interpolating solution. Gradient descent with small step size does this implicitly; other optimizers or large step sizes may not. Without this implicit regularization, large models can overfit catastrophically with no second descent.

3. Over-generalizing from clean data experiments: Double descent is most pronounced with label noise. On clean, well-structured data, the peak at the interpolation threshold can be subtle or absent. If your dataset has minimal noise, don't expect a dramatic double descent curve — and don't conclude the phenomenon doesn't exist just because you don't see it on MNIST.

4. Treating the interpolation threshold as a hard wall: The "threshold" isn't exactly at $N_{\text{params}} = N_{\text{train}}$ in practice. Effective model complexity (what the model actually uses, not parameter count), regularization (explicit or implicit), and data structure all shift the peak. The threshold is a useful conceptual guide, not a precise prediction.

Next Steps

25-05 — Mode Connectivity and Loss Landscapes — where you'll visualize the geometry of the loss landscape that double descent and grokking navigate, and understand why minima are connected by paths of low loss.