📐 Concept diagram

25-01 — Mechanistic Interpretability

Phase: 25 — Frontiers & Active Research Areas Subject: 25-01 Prerequisites: Phase 18–19 (LLM Math), Phase 14 (Optimization) Next subject: 25-02 — Sparse Autoencoders (SAEs)

Learning Objectives

By the end of this subject, you will be able to:

1. Define mechanistic interpretability and distinguish it from behavioral interpretability
2. Explain the superposition hypothesis and its implications for neural network representations
3. Understand features, circuits, and the distinction between polysemantic and monosemantic neurons
4. Apply causal scrubbing to validate mechanistic hypotheses about model behavior
5. Connect mechanistic interpretability to the broader agenda of AI alignment and safety

Core Content

What Is Mechanistic Interpretability?

Mechanistic interpretability is the study of reverse-engineering neural networks into human-understandable algorithms. Unlike behavioral interpretability (which treats the model as a black box and probes input-output relationships), mechanistic interpretability aims to understand the internal computations — the circuits of weights and activations that implement specific behaviors.

The central question: Can we decompose a neural network's computation into components we understand, in the same way we can decompile a binary into source code?

⚠️ CRITICAL: Mechanistic interpretability is fundamentally an empirical science of a mathematical object. A trained neural network is a deterministic function defined by its architecture and weights. There is a ground-truth answer to "how does this model compute output $y$ from input $x$?" — we just don't know how to extract it yet.

Features and Directions

A feature is a human-interpretable property of the input that the model represents internally. In the linear representation hypothesis, features correspond to directions in activation space:

$$\text{feature}_i(\mathbf{x}) = \mathbf{w}_i^T \cdot \mathbf{a}(\mathbf{x})$$

where $\mathbf{a}(\mathbf{x}) \in \mathbb{R}^d$ is the activation vector at some layer for input $\mathbf{x}$, and $\mathbf{w}_i \in \mathbb{R}^d$ is the feature direction.

Key empirical finding: In transformer language models, many interpretable features exist as nearly-linear directions. For example, you can find a direction that encodes "is the text in French" and intervene by adding/subtracting it.

The Superposition Hypothesis

The superposition hypothesis (Anthropic, 2022) posits that neural networks represent more features than they have dimensions, by exploiting a mathematical fact: sparse vectors in high dimensions can be almost-orthogonal.

If a model has $d$ neurons in a layer, classical thinking says it can represent at most $d$ independent features. But if features are sparse (only a few are active at once), the model can cram $n \gg d$ features into $d$-dimensional space using almost-orthogonal vectors.

Why this works: In $\mathbb{R}^d$, you can pack exponentially many almost-orthogonal vectors — the Johnson-Lindenstrauss lemma tells us there exist $\exp(O(d))$ vectors whose pairwise dot products are bounded by $\varepsilon$. When features are sparse, interference (dot products between active features) is small, even if the vectors aren't perfectly orthogonal.

Mathematical model of superposition (toy setting):

Given $n$ features with sparsity $S$ (fraction active), we embed them into $\mathbb{R}^d$ with $d < n$:

$$\mathbf{a}(\mathbf{x}) = \mathbf{W}^T \mathbf{f}(\mathbf{x}) \quad \text{where} \quad \mathbf{W} \in \mathbb{R}^{n \times d}, \ |\mathbf{w}_i| = 1$$

The reconstruction of feature $i$ from the activation is:

$$\hat{f}_i(\mathbf{x}) = \text{ReLU}(\mathbf{w}_i \cdot \mathbf{a}(\mathbf{x}) + b_i)$$

The model learns $\mathbf{W}$ to minimize reconstruction error subject to the bottleneck $d < n$. The optimal arrangement pushes feature vectors to be near-orthogonal, with interference manifesting as small-but-nonzero dot products.

Polysemanticity emerges naturally from superposition: when a single neuron participates in multiple feature directions, it responds to multiple seemingly-unrelated concepts. A polysemantic neuron activates for several distinct features; a monosemantic neuron activates primarily for one.

Circuits

A circuit is a subgraph of the neural network's computational graph that implements a specific behavior. Formally, a circuit $\mathcal{C}$ for behavior $B$ is a subset of edges (weights) such that:

$$\text{Model}(x){\text{with } \mathcal{C} \text{ ablated}} \neq \text{Model}(x){\text{full}} \quad \text{for inputs relevant to } B$$

while ablating edges outside $\mathcal{C}$ preserves behavior $B$.

Example: Induction heads (Olsson et al., 2022). In transformers, a two-layer circuit composed of a "previous token head" and a "copy head" implements the pattern-matching behavior: "If I see [A][B]..., then [A] again, predict [B]." Mathematically:

$$\text{Induction}(x_{1:t}) = \text{softmax}\left(\frac{Q_{\text{prev}} \cdot K_{\text{copy}}^T}{\sqrt{d_k}}\right) V_{\text{token}}$$

The K-composition from previous-token head feeds into the Q of the copy head, creating a circuit that attends to the token after each occurrence of the current token.

Universality Hypothesis

The universality hypothesis states that neural networks trained on similar data/tasks converge to similar internal circuits, regardless of random initialization. Evidence includes:

1. Feature universality: The same interpretable features appear across different training runs
2. Circuit universality: Induction heads, copy mechanisms, and other circuits emerge consistently
3. Representation similarity metrics: CKA (Centered Kernel Alignment) shows high similarity between different runs

Mathematically, if two models $f_\theta$ and $f_\phi$ are trained on the same distribution, there often exists an orthogonal transformation $R$ such that activations align: $\mathbf{a}\theta(\mathbf{x}) \approx R \cdot \mathbf{a}\phi(\mathbf{x})$.

Causal Scrubbing

Causal scrubbing is a methodology for rigorously testing mechanistic hypotheses. The idea:

1. Hypothesize a circuit $\mathcal{C}$ that implements behavior $B$
2. Scrub: Replace activations in the model with activations from different inputs that should be equivalent under the hypothesis
3. Measure: If the hypothesis is correct, the scrubbed model should produce the same output

Formally, given a hypothesis $H$ that identifies which activations encode what information, a scrubbing function $s: \mathcal{X} \to \mathcal{X}$ maps inputs to "equivalent" inputs. For each activation $a$ hypothesized to encode information $I(a)$, we replace it with the activation from $s(x)$ which (under $H$) encodes the same $I(a)$.

$$\text{Scrubbed output} = \text{Model}_{\text{with replaced activations}}(x)$$

If $H$ is correct: $\text{Model}(x) \approx \text{Scrubbed output}$.

Key Terms

• Causal scrubbing
• Circuits
• Hypothesize
• Measure
• Mechanistic interpretability
• Polysemantic neurons
• Polysemanticity
• Scrub

Worked Examples

Example 1: Finding a Feature Direction via Linear Probing

Problem: In a sentiment-classification transformer, you suspect a direction encodes "positive sentiment." The model's last-layer activations for two sentences are:

• "I love this" → $\mathbf{a}_1 = [0.8, -0.3, 0.5]^T$
• "I hate this" → $\mathbf{a}_2 = [-0.6, 0.2, -0.4]^T$

Find a candidate feature direction $\mathbf{w} \in \mathbb{R}^3$ that separates these.

Solution: 1. The vector difference points from negative to positive: $\mathbf{d} = \mathbf{a}_1 - \mathbf{a}_2 = [1.4, -0.5, 0.9]^T$ 2. Normalize: $|\mathbf{d}| = \sqrt{1.96 + 0.25 + 0.81} = \sqrt{3.02} \approx 1.738$ 3. $\mathbf{w} = \mathbf{d} / |\mathbf{d}| \approx [0.805, -0.288, 0.518]^T$ 4. Verify: $\mathbf{w}^T\mathbf{a}_1 \approx 0.644 + 0.086 + 0.259 = 0.989$ (positive) 5. $\mathbf{w}^T\mathbf{a}_2 \approx -0.483 - 0.058 - 0.207 = -0.748$ (negative) 6. The direction cleanly separates the two sentiments.

Example 2: Detecting Polysemanticity

Problem: A neuron has the following top-5 activating dataset examples: 1. "The cat sat on the mat" (token "cat" → activation 8.2) 2. "Heavy metal music is loud" (token "metal" → activation 7.9) 3. "The bank of the river" (token "bank" → activation 8.5) 4. "I need to deposit money at the bank" (token "bank" → activation 8.3) 5. "My favorite cat breed is Siamese" (token "cat" → activation 7.8)

Is this neuron monosemantic or polysemantic?

Solution: The neuron activates strongly for: "cat" (animal), "metal" (music genre), and "bank" (both riverbank and financial bank). These concepts are semantically unrelated — there's no single concept that encompasses "cats, heavy metal music, and riverbanks." Therefore, this neuron is polysemantic, representing at least 3-4 distinct features in superposition.

Example 3: Causal Scrubbing of an Induction Circuit

Problem: You hypothesize that attention head L1.H3 implements the pattern $[A] \to [B]$ (attend to the token after the current token's last occurrence). Design a scrubbing experiment to test this. Given the input sequence "The cat sat on the mat. The cat", the model predicts "sat". What would scrubbing look like?

Solution: 1. Hypothesis: L1.H3 attends from "cat" (position 6) to the token after the previous "cat" (position 1), which is "sat" at position 2. The value at position 2 is then used to predict "sat." 2. Scrubbing design: Replace the activations of L1.H3 at position 6 with activations from a different input where the hypothesis predicts the same behavior. For instance, consider "A dog ran in the park. The dog" — here L1.H3 would similarly attend from "dog" (pos 6) to "ran" (pos 2). If L1.H3's output encodes "the token after the previous identical token," scrubbing should preserve behavior. 3. Execution: Run the model twice. First, normally on "The cat sat on the mat. The cat." Second, with L1.H3's output at position 6 replaced by L1.H3's output from "A dog ran...the dog" at the corresponding position. 4. Prediction under correct hypothesis: The model should still predict "sat" because L1.H3's output from the dog example encodes the same algorithmic signal (attend to the token after the match), and "sat" is retrieved from the unscrubbed residual stream. 5. If hypothesis is wrong: The prediction would change because the replaced activation encodes different information than hypothesized.

Practice Problems

Problem 1: A 3-dimensional activation space has features $\mathbf{f}_1 = [1, 0, 0]^T$, $\mathbf{f}_2 = [0, 1, 0]^T$, $\mathbf{f}_3 = [0, 0, 1]^T$, and $\mathbf{f}_4 = [0.6, 0.6, 0.528]^T$. Compute all pairwise dot products and determine if feature 4 can coexist with the others in superposition with minimal interference.

Problem 2: In a probing experiment, you train a linear classifier to predict whether a sentence contains a negation from a model's layer-3 activations. The probe achieves 99% accuracy. Does this prove the model uses a negation feature for its predictions? Why or why not?

Problem 3: Suppose you discover that two attention heads, H1 and H2, form a circuit for factual recall from a subject-relation-object triplet. Describe the minimum evidence needed to claim you've "understood" this circuit.

Problem 4: A layer has dimensionality $d = 512$ and represents features that are each active with probability $p = 0.01$ (sparsity $S=0.01$, or 1% of features active per input). Estimate the maximum number of features that can be packed into this layer, assuming features are random unit vectors in $\mathbb{R}^d$ and interference $\geq 0.5$ becomes problematic.

Problem 5: Causal scrubbing: You hypothesize that a neuron N encodes "the subject is plural." Design two scrubbing experiments — one that should pass if your hypothesis is correct, and one that should fail (as a control).

Summary

• Mechanistic interpretability reverse-engineers neural networks into human-understandable algorithms — finding features, circuits, and the computations they implement
• The superposition hypothesis explains how networks represent $n \gg d$ features in $d$-dimensional space using almost-orthogonal vectors and sparsity
• Polysemantic neurons fire for multiple unrelated concepts because they participate in multiple feature directions; monosemantic neurons respond to a single concept
• Circuits are subgraphs of weights that implement specific behaviors; understanding them involves proving necessity, sufficiency, and composition
• Causal scrubbing is the gold standard for testing mechanistic hypotheses — it replaces activations across different inputs and checks if behavior is preserved
• The universality hypothesis suggests different training runs converge to similar circuits, implying underlying algorithmic structure is not random

Quiz

Question 1: What is the superposition hypothesis?

A. Neural networks can represent at most $d$ features in a $d$-dimensional layer B. Neural networks can represent $n > d$ features in $d$ dimensions by exploiting sparsity and near-orthogonality C. All features in a network are orthogonal to each other by construction D. Superposition only occurs in attention layers, not MLP layers

Correct Answer: B

Question 2: A neuron fires strongly for tokens "dog," "puppy," and "canine" — and nothing else. This neuron is:

A. Polysemantic B. Monosemantic C. Dead D. A superposition artifact

Correct Answer: B

Question 3: In causal scrubbing, what does it mean if scrubbing an activation does NOT change the model's output?

A. The activation is irrelevant to the model's computation B. The scrubbing function was incorrectly designed C. The hypothesis about what the activation encodes is supported (the activation was correctly replaced with equivalent information) D. The model is too robust to be interpretable

Correct Answer: C

Question 4: The universality hypothesis suggests:

A. Every training run produces a completely different set of internal circuits B. Neural networks from different architectures always converge to identical weights C. Models trained on similar data tend to develop similar internal features and circuits across different random seeds D. Interpretability methods work equally well on all architectures

Correct Answer: C

Question 5: Which of these is an example of a circuit in mechanistic interpretability?

A. A single neuron that fires for cat images B. A learned weight matrix in an attention layer C. A specific subgraph of attention heads and MLP neurons that together implement indirect object identification D. The entire forward pass of the model

Correct Answer: C

Question 6: Why is linear probing alone insufficient to establish that a model uses a feature?

A. Linear probes are computationally too expensive B. Linear probes can detect information that is present but not causally used by the model C. Linear probes only work on convolutional networks D. Linear probes always overfit

Correct Answer: B

Pitfalls

1. Confusing correlation with causation: Finding a direction that encodes information ≠ the model uses that information. Always follow probing with causal experiments.
2. Over-interpreting individual neurons: In the superposition regime, single neurons are entangled. Looking at neurons in isolation gives an incomplete picture.
3. Assuming perfect orthogonality: Feature vectors are almost orthogonal, not perfectly orthogonal. The small dot products create interference patterns that matter.
4. Neglecting the residual stream: In transformers, the residual stream is the primary communication channel. Many interpretability efforts focus too narrowly on attention patterns.

Pitfalls

1. Confusing correlation with causation in probing experiments: Just because a linear probe can decode "French" from layer 5 activations doesn't mean the model uses French representations for its own predictions. Always follow probing with causal intervention (activation patching, ablation). This is the single most common error in interpretability research.

2. Focusing exclusively on attention patterns: While attention patterns are visually interpretable, the residual stream and MLP layers carry substantial computation. Many circuits involve MLP neurons that transform information in ways attention patterns alone can't capture. Ignoring MLPs gives an incomplete picture of model computation.

3. Over-relying on neuron-level analysis in the superposition regime: When features are in superposition, individual neurons are polysemantic — they respond to mixtures of features. Analyzing neurons one at a time, without considering their interactions, can lead to misleading conclusions about what the model "knows." Use dictionary learning (SAEs, 25-02) to disentangle features.

4. Neglecting the residual stream as the primary communication channel: In transformers, attention heads and MLPs write to and read from the residual stream. It's the shared memory. Many interpretability efforts treat attention outputs as directly used by the next layer, missing that the residual stream accumulates information across layers. Circuits often span the residual stream, not just attention-to-attention connections.

Next Steps

25-02 — Sparse Autoencoders (SAEs) — where you'll learn how to automatically discover interpretable features from activations, providing a key tool for the mechanistic interpretability agenda.