📐 Concept diagram

25-09 — Continual Learning

Phase: 25 — Frontiers & Active Research Areas Subject: 25-09 Prerequisites: 16 (Neural Networks), 24-01 (Fisher Information), 20 (Training & Fine-tuning) Next subject: 25-10 — Multimodal Models Mathematics

Learning Objectives

By the end of this subject, you will be able to:

1. Define catastrophic forgetting and explain why standard SGD causes it
2. Derive Elastic Weight Consolidation (EWC) from a Bayesian perspective using the Laplace approximation
3. Implement experience replay and explain the stability-plasticity dilemma
4. Compare regularisation-based, replay-based, and architectural approaches to continual learning
5. Analyse the trade-offs between forward and backward transfer in continual learning benchmarks

Core Content

1. The Catastrophic Forgetting Problem

When a neural network is trained sequentially on multiple tasks, learning task $B$ degrades performance on previously-learned task $A$ — sometimes catastrophically, dropping to chance level. This is catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999).

⚠️ CRITICAL: Standard SGD with no continual learning mechanisms optimises only the current task's loss. The parameter updates that reduce loss on task $B$ typically move parameters out of the region where task $A$'s loss is low — there is no mechanism to preserve prior knowledge.

The stability-plasticity dilemma (Grossberg, 1987): - Stability: The ability to retain knowledge of previous tasks - Plasticity: The ability to learn new tasks quickly

A system needs both — too much stability prevents learning, too much plasticity causes forgetting. Continual learning algorithms navigate this trade-off.

Formal problem statement: Given a sequence of tasks $\mathcal{T}1, \mathcal{T}_2, \ldots, \mathcal{T}_T$, each with its own data distribution $P_t(\mathbf{x}, y)$, learn a model $f\theta$ that performs well on all tasks seen so far, without storing all previous data and without knowing task boundaries in advance.

Continual learning comes in three scenarios: - Task-incremental: Task identity is known at both training and test time (easiest) - Domain-incremental: Task identity is NOT known at test time, but the output structure is the same - Class-incremental: New classes are added over time — the model must distinguish all classes seen so far

2. Elastic Weight Consolidation (EWC)

EWC (Kirkpatrick et al., 2017) is a regularisation-based approach that penalises changes to parameters that are important for previous tasks.

Bayesian derivation: After learning task $A$, the posterior over parameters is:

$$\log p(\boldsymbol{\theta} | D_A) = \log p(D_A | \boldsymbol{\theta}) + \log p(\boldsymbol{\theta}) - \text{const}$$

For task $B$, the posterior given both datasets is:

$$\log p(\boldsymbol{\theta} | D_A, D_B) = \log p(D_B | \boldsymbol{\theta}) + \log p(\boldsymbol{\theta} | D_A) - \text{const}$$

The term $\log p(\boldsymbol{\theta} | D_A)$ is intractable — EWC approximates it with a Gaussian centred at the task $A$ optimum $\boldsymbol{\theta}_A^*$, with precision given by the Fisher information matrix:

$$p(\boldsymbol{\theta} | D_A) \approx \mathcal{N}(\boldsymbol{\theta}_A^*, F^{-1})$$

where $F = \mathbb{E}{(\mathbf{x},y)\sim D_A}[(\nabla\theta \log p(y|\mathbf{x}, \boldsymbol{\theta}_A^))(\nabla_\theta \log p(y|\mathbf{x}, \boldsymbol{\theta}_A^))^\top]$ is the Fisher.

This is a Laplace approximation — the log-posterior is Taylor-expanded around the mode to second order, and the Hessian is approximated by the Fisher.

EWC loss for task $B$:

$$L_{\text{EWC}}(\boldsymbol{\theta}) = L_B(\boldsymbol{\theta}) + \frac{\lambda}{2} \sum_i F_i (\theta_i - \theta_{A,i}^*)^2$$

where $F_i$ is the $i$-th diagonal entry of the Fisher (for computational tractability, the diagonal approximation is used). Parameters with high Fisher information (important for task $A$) are heavily penalised from moving.

Extending to $T$ tasks: Maintain the Fisher diagonals and optimal parameters for all previous tasks, or use online EWC which maintains a single running Fisher and reference parameter set.

3. Experience Replay

Experience replay stores a small subset of previous-task data and interleaves it with new-task data during training.

Reservoir sampling: Maintain a memory buffer $\mathcal{M}$ of fixed size $M$. For each new example, with probability $M/(\text{seen examples})$, replace a random existing example.

Training with replay: On each SGD step, sample a mini-batch that mixes current-task data with replay data:

$$L(\boldsymbol{\theta}) = \mathbb{E}{(\mathbf{x},y)\sim D{\text{current}}}[L(f_\theta(\mathbf{x}), y)] + \alpha \cdot \mathbb{E}{(\mathbf{x},y)\sim \mathcal{M}}[L(f\theta(\mathbf{x}), y)]$$

The replay term prevents forgetting by keeping the loss on previous tasks in the optimisation objective.

⚠️ Trade-off: Replay is simple and effective but requires storing data — which may violate privacy constraints or memory limitations. Regularisation methods like EWC avoid storing data but are generally less effective at preventing forgetting.

4. Progressive Neural Networks

Instead of modifying a single model, progressive networks (Rusu et al., 2016) add new capacity for each task:

• Column $k$ is a neural network trained on task $k$
• Column $k$ receives lateral connections from all previous columns ${1, \ldots, k-1}$:

$$h_i^{(k)} = f\left(W_i^{(k)} h_{i-1}^{(k)} + \sum_{j < k} U_i^{(k,j)} h_{i-1}^{(j)}\right)$$

• Previous columns are frozen — no forgetting by construction
• New columns adapt to new tasks using knowledge from previous tasks via lateral connections

Advantage: Zero forgetting — all previous knowledge is preserved. Disadvantage: Linear growth in parameters with the number of tasks.

5. Other Approaches

Synaptic Intelligence (SI): Similar to EWC but estimates parameter importance online during training by accumulating the contribution of each parameter to the loss decrease — no Fisher computation needed.

Memory Aware Synapses (MAS): Estimates importance by measuring the sensitivity of the learned function to parameter changes — $\Omega_i = \mathbb{E}{\mathbf{x}\sim D}[|\partial f\theta(\mathbf{x})/\partial \theta_i|]$, accumulated during training.

Learning without Forgetting (LwF): Uses knowledge distillation — the old model's predictions on new-task data serve as soft targets, encouraging the new model to preserve old-task responses.

Key Terms

• Average accuracy
• Catastrophic forgetting
• Experience replay
• Laplace approximation
• Progressive networks

Worked Examples

Example 1: EWC Weight Importance

Problem: A 3-parameter model has been trained on task A. The Fisher diagonals are $F = [0.01, 5.2, 0.3]$ and optimal parameters $\boldsymbol{\theta}_A^* = [1.0, -0.5, 2.0]$. During task B training, the SGD update (without EWC) would change parameters to $[1.2, 2.1, 1.8]$. Compute the EWC penalty and the effective gradient correction for $\lambda = 10$.

Solution:

EWC penalty: $L_{\text{EWC}} = \frac{\lambda}{2}\sum_i F_i(\theta_i - \theta_{A,i}^*)^2$

At the proposed point: - $(1.2-1.0)^2 \cdot 0.01 \cdot 5 = 0.04 \cdot 0.05 = 0.002$ - $(2.1-(-0.5))^2 \cdot 5.2 \cdot 5 = 6.76 \cdot 26 = 175.8$ - $(1.8-2.0)^2 \cdot 0.3 \cdot 5 = 0.04 \cdot 1.5 = 0.06$

Total penalty = 175.9 — dominated by parameter 2, whose Fisher (5.2) indicates high importance. The gradient of the EWC penalty for parameter 2 is $\lambda F_2(\theta_2 - \theta_{2,A}^*) = 10 \cdot 5.2 \cdot 2.6 = 135.2$, strongly opposing the proposed change.

Parameter 1 (Fisher 0.01) is essentially free to move — the penalty gradient is only $10 \cdot 0.01 \cdot 0.2 = 0.02$.

Example 2: Progressive Network Lateral Connections

Problem: A progressive network has 2 columns. Column 1 (trained on MNIST) has hidden layer $h_1^{(1)} = \text{ReLU}(W_1^{(1)}\mathbf{x})$. Column 2 (trained on Fashion-MNIST) receives lateral input from column 1. If $W_1^{(2)} \in \mathbb{R}^{128 \times 784}$ and $U_1^{(2,1)} \in \mathbb{R}^{128 \times 64}$ where column 1's hidden size is 64, write the forward pass for column 2's first hidden layer and compute its parameter count.

Solution:

Forward pass: $h_1^{(2)} = \text{ReLU}(W_1^{(2)}\mathbf{x} + U_1^{(2,1)}h_1^{(1)})$

Parameter count: $W_1^{(2)}$ has $128 \times 784 = 100,352$ params, $U_1^{(2,1)}$ has $128 \times 64 = 8,192$ params. Total for this layer: 108,544. The lateral connection adds only 8% overhead compared to the standard weight matrix, while providing access to column 1's learned features. Note that $W_1^{(1)}$ is NOT counted — it's frozen.

Example 3: Stability-Plasticity Measurement

Problem: After sequential training on tasks $A \to B \to C$, evaluate the model on all three tasks. Results: $A = 0.92 \to 0.31 \to 0.28$, $B = \text{N/A} \to 0.88 \to 0.42$, $C = \text{N/A} \to \text{N/A} \to 0.85$. Compute the forgetting measure for task A after learning task B, and for task B after learning task C.

Solution:

Forgetting = (best performance on task) − (current performance)

Task A after B: $0.92 - 0.31 = 0.61$ (61% forgetting — catastrophic) Task A after C: $0.92 - 0.28 = 0.64$ (64% total forgetting) Task B after C: $0.88 - 0.42 = 0.46$ (46% forgetting)

Average forgetting = $(0.64 + 0.46)/2 = 0.55$ — the model forgets over half its prior knowledge. Forward transfer from A→B: $B_{\text{with A}}/B_{\text{scratch}}$ — not computable without scratch performance.

Practice Problems

Problem 1: Derive the EWC regularisation term from the Bayesian posterior. Show how the Laplace approximation converts $\log p(\boldsymbol{\theta}|D_A)$ into the quadratic penalty.

Problem 2: Compare the memory cost of experience replay (storing $M$ examples) vs EWC (storing Fisher diagonals). For ImageNet-sized models (25M parameters), which is more memory-efficient?

Problem 3: Explain why progressive networks avoid catastrophic forgetting entirely. What is the cost?

Problem 4: The online EWC algorithm merges Fisher matrices: $\tilde{F}t = \gamma \tilde{F}{t-1} + F_t$, where $F_t$ is the Fisher after task $t$. Explain the role of $\gamma < 1$ and what happens as $t \to \infty$.

Problem 5: Design a continual learning benchmark that distinguishes between a method that genuinely retains knowledge and one that simply learns a single representation that works for all tasks. What metrics would you use?

Summary

1. Catastrophic forgetting is the fundamental challenge of continual learning — new task training overwrites parameters crucial for old tasks.
2. EWC uses a Bayesian Laplace approximation to penalise changes to parameters with high Fisher information — it estimates parameter importance from the curvature of the loss.
3. Experience replay stores and interleaves past data, providing direct supervision on old tasks — effective but requires memory and may violate privacy.
4. Progressive networks eliminate forgetting by freezing old columns and adding new ones — zero forgetting at the cost of linear parameter growth.
5. The stability-plasticity dilemma governs all CL approaches — methods trade off knowledge retention against learning capacity.

Pitfalls

1. Diagonal Fisher approximation: EWC uses only diagonal entries of the Fisher matrix, ignoring off-diagonal interactions between parameters. This is computationally necessary but loses information about correlated parameter importance.
2. Task boundary knowledge: Many CL methods assume known task boundaries. In realistic scenarios, data arrives continuously without clear boundaries — this is the harder "task-free" continual learning setting.
3. Fixed memory replay: Reservoir sampling is unbiased but discards examples. Prioritised replay weights storage toward examples the model is most likely to forget, improving retention for the same memory budget.

Quiz

Question 1: Catastrophic forgetting occurs when:

A. The model runs out of GPU memory during training B. Learning a new task degrades performance on previously learned tasks, sometimes to chance level C. The learning rate is set too low for convergence D. The training dataset is too small to learn anything useful

Correct Answer: B

Question 2: Elastic Weight Consolidation (EWC) penalizes changes to parameters based on:

A. The absolute magnitude of each parameter B. The Fisher information matrix — parameters with high Fisher information (important for previous tasks) are heavily constrained from moving C. Random selection of parameters to freeze D. The initialization values of the parameters

Correct Answer: B

Question 3: Why does EWC use the diagonal approximation of the Fisher information matrix rather than the full matrix?

A. The diagonal is more accurate for measuring parameter importance B. The full Fisher is $O(P^2)$ in size for $P$ parameters — storing and inverting it is infeasible for modern models with millions of parameters C. All off-diagonal entries are exactly zero by definition D. The diagonal provides stronger regularization

Correct Answer: B

Question 4: Experience replay prevents catastrophic forgetting by:

A. Freezing all model parameters after each task B. Storing a subset of past-task data and interleaving it with current-task data during training, providing direct supervision on old tasks C. Adding L2 weight decay to the loss function D. Increasing the model size for each new task

Correct Answer: B

Question 5: Progressive neural networks achieve zero catastrophic forgetting by:

A. Storing all data from all previous tasks B. Freezing previously trained columns and adding new columns with lateral connections for each new task — previous parameters are never modified C. Using extremely high learning rates to rapidly encode new tasks D. Training all tasks simultaneously in one batch

Correct Answer: B

Question 6: The stability-plasticity dilemma refers to the trade-off between:

A. Model size and inference speed B. Retaining knowledge of previous tasks (stability) vs. learning new tasks effectively (plasticity) C. Training time and final test accuracy D. Batch size and learning rate selection

Correct Answer: B

Next Steps

Final subject: 25-10 — Multimodal Models Mathematics — the endpoint of the entire LLM Researcher Mathematics Curriculum.