Skip to content
LumenaryAcademy
Frontier 75 min

Interpretability

Looking inside networks — probes, attention patterns, the residual stream, superposition, sparse autoencoders, circuits, patching and steering — and an honest account of what we still cannot see.

Code tested · 7AI-draftedAwaiting review by a computer scientist

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Why look inside?

Everything in the previous lessons treated a model from the outside: we trained it, sampled from it, and measured its behaviour (Evaluating models). Behavioural evaluation is indispensable, but it has a ceiling. It tells you what a model did on the inputs you tried, not why, and so not what it will do on inputs you didn't try. Two models with identical test scores can get there by different routes — one by a general method, one by a brittle shortcut — and the difference shows only off-distribution.

Interpretability tries to understand the computations inside a network: what information is represented where, and how it is transformed into outputs. The motivations range from the practical to the profound:

  • Debugging: find why a model fails, and fix the cause rather than the symptom.
  • Trust and auditing: check that a model is using legitimate reasons — not a spurious correlation, not a hidden objective.
  • Safety: detect deception or dangerous knowledge that behaviour might conceal, and verify that safety training changed what we think it changed (AI safety and alignment).
  • Control: edit or steer behaviour by intervening on internal representations.
  • Science: neural networks are a new kind of object that learned to do things we cannot write programs for; understanding how is interesting in its own right.

The subfield that tries to reverse-engineer networks into human-understandable algorithms is called mechanistic interpretability. This lesson tours its main tools, from the oldest and simplest to the current frontier, and ends with an honest account of its limits.

DefinitionsH

Probing: is the information there?

Definition— Probe

A probe is a simple classifier — usually linear — trained to predict some property of the input (a part of speech, whether a statement is true, the sentiment of a review) from a model's internal activations at a particular layer and position. High probe accuracy means the property is decodable from those activations.

Probes are cheap and informative (Alain and Bengio, 2016). Researchers have found linearly decodable representations of syntax, of the board state in models trained only on game move sequences, of spatial and temporal information, and of whether the model "thinks" a statement is true (Burns et al., 2022).

But a probe answers a narrow question, and it is easy to over-read:

  • Decodable is not used. A property may be present in the activations as a by-product and never influence the output. Only an intervention — changing the representation and seeing the behaviour change — shows that the model uses it.
  • A powerful probe can learn the task itself. A deep enough probe can extract nearly anything from rich activations. Hewitt and Liang (2019) proposed control tasks — random labels with the same structure — and reporting selectivity, the gap between the real task and the control. Linear probes, for this reason, are the norm.
  • Correlated properties. A "truth" probe might really detect "statements that sound like textbook sentences".

You will write a linear probe in the exercises. Keep in mind throughout: correlation from probes, causation from interventions.

ExplanationH

Attention patterns, and what they don't tell you

Attention weights are right there in the model, one probability distribution per head, query and layer (Attention and the transformer). Visualising them was among the first interpretability methods for transformers, and some patterns are clean and meaningful. Previous-token heads attend to the token just before the current one. Induction heads (Olsson et al., 2022) implement a simple, powerful algorithm: if the current token is AA, find an earlier occurrence of AA and predict whatever followed it ("…Ngozi Okonjo… Ngozi" → "Okonjo"). They work together with previous-token heads, appear abruptly during training at the same moment as a jump in the model's ability to learn from context, and are found across many models.

Yet attention weights are a weak explanation in general:

  • The output of a head is ∑jwjvj\sum_j w_j v_j. A large weight on a token whose value vector is near zero contributes nothing; the weights alone ignore the values.
  • Each layer's heads read a residual stream that already mixes information from many positions, so "attending to token 5" in layer 20 may really mean attending to information that arrived at position 5 from elsewhere.
  • Jain and Wallace (2019) found that quite different attention distributions could produce nearly the same predictions; Wiegreffe and Pinter (2019) replied that this depends on how the question is posed. The debate's lasting lesson: treat attention maps as hypotheses to test with interventions, not as conclusions.
IntuitionH

The residual stream: a shared workspace

The most productive way to think about a transformer's internals comes from Elhage et al. (2021). Each position carries a vector, the residual stream. Every attention head and MLP reads from the stream (through a linear projection), computes something, and adds its result back (the transformer block). Nothing is overwritten; everything is a sum. The final layer reads the stream and turns it into next-token logits through the unembedding matrix.

Three consequences:

  1. Components communicate through subspaces. Because the stream is a sum, a later component can read what an earlier one wrote if their projections align. Much of the mechanistic picture is linear algebra on these read and write matrices (Linear algebra).
  2. Intermediate predictions are visible. Apply the final unembedding directly to the stream at an intermediate layer — the logit lens, popularised in a 2020 blog post — and you see what the model "would predict" if it stopped there. The predictions sharpen layer by layer. The tuned lens learns a small correction per layer, which makes this much more reliable.
  3. The linear representation hypothesis. Many concepts appear to be represented as directions in activation space: an activation vector is, roughly, a sum of feature directions, each scaled by how strongly that feature is present. This is a hypothesis, supported by much evidence and contradicted in places (some features turn out to be multi-dimensional, such as days of the week arranged on a circle), but it is the working assumption behind most current tools.

If features are directions, why not just look at the neurons — the coordinates? In vision models some neurons do respond to one clean concept, such as curves of a particular orientation (Olah et al., 2020). But in language models most neurons are polysemantic: in one small transformer Anthropic studied, a single neuron responded to academic citations, Korean text and HTTP requests, among other things. The reason appears to be superposition.

ComputationC

Superposition: more features than dimensions

A model has a residual stream of a few thousand dimensions, yet seems to represent far more distinct concepts than that. How?

Two geometric facts make it possible. First, in high dimensions you can fit many more than dd directions that are nearly orthogonal to each other — exponentially many, if you tolerate small overlaps. Second, most features are sparse: any given token involves only a handful of the millions of concepts a model knows. If only a few features are active at once, their small overlaps rarely add up to confusion. Toy Models of Superposition showed that small networks trained on sparse synthetic features really do learn to store more features than they have dimensions, arranged in surprisingly regular geometric patterns, and that they stop doing so when features are dense.

python
import numpy as np

rng = np.random.default_rng(0)
d = 64                                            # dimensions of the "residual stream"

def random_directions(n, d):
    F = rng.normal(size=(n, d))
    return F / np.linalg.norm(F, axis=1, keepdims=True)

# 1. Far more than d directions can be *nearly* orthogonal in d dimensions.
for n in [64, 512, 2048]:
    F = random_directions(n, d)
    cos = np.abs(F @ F.T)[~np.eye(n, dtype=bool)]
    print(f"{n:5d} random directions in {d} dims: mean |cos| {cos.mean():.2f}, 99th percentile {np.quantile(cos, 0.99):.2f}")

# 2. Store 512 features in 64 dimensions. Activate a few, add their directions,
#    then try to read off which ones were active using dot products alone.
F = random_directions(512, d)
print()
for k in [1, 3, 10, 30]:
    hits = 0
    for _ in range(300):
        active = rng.choice(512, size=k, replace=False)
        x = F[active].sum(axis=0)                 # the activation vector: a sum of feature directions
        guess = np.argsort(-(F @ x))[:k]          # the k features best aligned with x
        hits += len(set(guess) & set(active))
    print(f"{k:2d} active features out of 512: {hits / (300 * k):.0%} recovered by simple readout")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The typical overlap between random directions does not grow as you add more of them: 2,048 directions in 64 dimensions are as nearly orthogonal, on average, as 64. And eight times more features than dimensions can be stored and read back almost perfectly — as long as only a few are active at once. As more are active, interference swamps the readout. That is the bargain superposition strikes: capacity in exchange for occasional interference, a good deal when features are sparse.

It is also bad news for interpretation. If features are directions that don't line up with neurons, then looking at neurons shows us mixtures. We need a way to find the directions.

ComputationC

Sparse autoencoders: finding the features

If activations are sparse combinations of feature directions, then recovering the features is a dictionary learning problem, studied for decades in signal processing and neuroscience: find an overcomplete set of directions (the dictionary) such that each activation vector is a sparse, non-negative combination of a few of them.

Definition— Sparse autoencoder (SAE)

Given model activations x∈Rdx \in \mathbb{R}^d, an SAE with m≫dm \gg d latents computes

f=ReLU⁡(Wenc(x−bdec)+benc)∈Rm,x^=Wdecf+bdec,f = \operatorname{ReLU}\big(W_{\text{enc}}(x - b_{\text{dec}}) + b_{\text{enc}}\big) \in \mathbb{R}^m, \qquad \hat x = W_{\text{dec}} f + b_{\text{dec}},

and is trained to minimise ∥x−x^∥2+λ∥f∥1\lVert x - \hat x \rVert^2 + \lambda \lVert f \rVert_1. The columns of WdecW_{\text{dec}} (kept at unit norm) are the learned feature directions; ff says how strongly each is active. The L1L_1 penalty pushes most of ff to zero.

Here we build data with known ground truth — 30 sparse features in superposition in 12 dimensions — and see whether an SAE can find them:

python
import numpy as np

rng = np.random.default_rng(0)
d, n_true, n_samples = 12, 30, 20000
# Ground truth: 30 sparse, non-negative "features", stored in superposition in 12 dimensions.
true_dirs = rng.normal(size=(n_true, d))
true_dirs /= np.linalg.norm(true_dirs, axis=1, keepdims=True)
active = rng.random((n_samples, n_true)) < 0.04                   # each feature is rarely on
X = (active * rng.uniform(0.5, 1.5, size=(n_samples, n_true))) @ true_dirs   # what we get to see

def train_sae(lam, m=30, steps=3000, lr=3e-3):
    """Sparse autoencoder: f = ReLU(W_enc (x - b_dec) + b_enc), x_hat = W_dec f + b_dec,
    loss = ||x - x_hat||^2 + lam * ||f||_1, trained with Adam; gradients written out by hand."""
    W_enc = rng.normal(scale=0.3, size=(m, d)); b_enc = np.zeros(m)
    W_dec = W_enc.T.copy(); b_dec = np.zeros(d)
    params = [W_enc, b_enc, W_dec, b_dec]
    m1 = [np.zeros_like(p) for p in params]; m2 = [np.zeros_like(p) for p in params]
    for step in range(1, steps + 1):
        x = X[rng.integers(0, n_samples, 256)]
        pre = (x - b_dec) @ W_enc.T + b_enc
        f = np.maximum(pre, 0)
        err = f @ W_dec.T + b_dec - x
        g_xhat = 2 * err / len(x)
        g_pre = (g_xhat @ W_dec + lam / len(x)) * (pre > 0)
        grads = [g_pre.T @ (x - b_dec), g_pre.sum(0), g_xhat.T @ f, g_xhat.sum(0) - g_pre.sum(0) @ W_enc]
        for p, g, a, b in zip(params, grads, m1, m2):
            a *= 0.9; a += 0.1 * g
            b *= 0.999; b += 0.001 * g * g
            p -= lr * (a / (1 - 0.9**step)) / (np.sqrt(b / (1 - 0.999**step)) + 1e-8)
        W_dec /= np.maximum(np.linalg.norm(W_dec, axis=0, keepdims=True), 1e-8)   # unit-norm dictionary
    rel_err = np.mean(err**2) / np.mean(x**2)
    l0 = (f > 0).sum(1).mean()
    recovered = int((np.abs(true_dirs @ W_dec).max(axis=1) > 0.9).sum())
    return rel_err, l0, recovered

print(f"true number of active features per input: {active.sum(1).mean():.1f}\n")
print(f"{'L1 weight':>9} {'recon. error':>12} {'active latents':>15} {'true features found':>20}")
for lam in [0.05, 0.4]:
    rel_err, l0, recovered = train_sae(lam)
    print(f"{lam:9.2f} {rel_err:12.3f} {l0:15.1f} {recovered:14d} of {n_true}")

# The 12 principal components cannot do this: there are only 12 of them, and they are orthogonal.
pcs = np.linalg.svd(X - X.mean(0), full_matrices=False)[2]
print("\nPCA directions matching a true feature (cosine > 0.9):", int((np.abs(true_dirs @ pcs.T).max(1) > 0.9).sum()))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With a weak sparsity penalty the SAE reconstructs almost perfectly but uses many latents per input and finds only a few true features: it has learned a good basis, not the features. With a stronger penalty, reconstruction is worse but the latents are sparse and most of the 30 true directions are found, though there are only 12 dimensions. PCA, limited to 12 orthogonal directions, finds none. That trade-off between reconstruction and sparsity, with interpretability hiding in the sparse regime, is exactly what practitioners tune on real models.

What it found in real models. Anthropic's Towards Monosemanticity (2023) applied SAEs to a small one-layer transformer and found thousands of features far more interpretable than its neurons: DNA sequences, Arabic script, base64, legal language. Scaling Monosemanticity (2024) scaled to a production model, Claude 3 Sonnet, with millions of features, including multilingual and multimodal ones for specific places and people, for bugs in code, for sycophantic praise, for deception and for dangerous topics. Clamping one feature, for the Golden Gate Bridge, produced a model that brought the bridge into every answer, a memorable demonstration that the features are causally used. Others reached similar results independently: Cunningham et al. on open models, OpenAI with SAEs of millions of latents on GPT-4 and an improved "TopK" design (Gao et al., 2024), and Google DeepMind's Gemma Scope, a public release of SAEs for every layer of two of its open Gemma 2 models, explorable on Neuronpedia.

Known problems. SAEs do not reconstruct activations perfectly, and the residual error — sometimes called dark matter — may hold important computation. The features found depend on the dictionary size: a "dog" feature in a small SAE can split into several in a larger one. Some latents never activate ("dead") or fire on almost everything. The L1L_1 penalty systematically shrinks activations, which newer designs (TopK, JumpReLU) address. And in 2025 the Google DeepMind interpretability team reported that, on a practical task they cared about (detecting harmful intent), probes built from SAE features did worse than plain linear probes, and announced they were shifting effort away from SAEs. Whether SAEs are the right foundation, a useful approximation, or a stepping stone is actively debated. Newer variants — transcoders, which approximate an MLP's input–output map with sparse features, and crosscoders, which share features across layers or models — try to fix some of these issues.

ComputationC

Circuits and activation patching

Features say what is represented. Circuits say how features are computed from one another: subgraphs of the network — particular heads, neurons or features and the connections between them — that implement a recognisable algorithm. Classic examples include curve detectors built from earlier edge detectors in vision models (Zoom In), induction heads, the circuit by which GPT-2 small completes sentences like "When Ayesha and Tomás went to the market, Tomás gave a book to" with "Ayesha" (Wang et al., 2022), and the modular-addition algorithm a small network learned using discrete Fourier transforms (Nanda et al., 2023).

The workhorse for finding circuits is an intervention.

Definition— Activation patching

Choose a clean input on which the model shows a behaviour, and a minimally different corrupted input on which it does not. Run the model on the corrupted input, but replace the activation of one component with the value it had on the clean run, and measure how much of the clean behaviour returns. Usually reported as the fraction restored,

mpatched−mcorruptmclean−mcorrupt,\frac{m_{\text{patched}} - m_{\text{corrupt}}}{m_{\text{clean}} - m_{\text{corrupt}}},

for a metric mm such as the difference between the logits of the correct and incorrect answers.

A component whose patch restores the behaviour is sufficient to carry the relevant information; patching in the other direction (clean run, corrupted activation) tests whether it is necessary. The method is also called causal tracing or interchange intervention. Here it is on a tiny network, together with steering, which we discuss next:

python
import numpy as np

rng = np.random.default_rng(0)
# A tiny two-layer network answering "what is the capital of <country>?" with Paris or Rome.
# Inputs: dims 0-1 encode the country, dims 2-7 are irrelevant context.
FRANCE, ITALY = np.array([1.0, 0.0]), np.array([0.0, 1.0])
def make_input(country):
    return np.concatenate([country, rng.normal(size=6)])

W1 = rng.normal(scale=0.3, size=(8, 8))
W1[0, :2] = [2.0, -1.0]                    # hidden unit 0 has learned "France-ness"
W1[1, :2] = [-1.0, 2.0]                    # hidden unit 1 has learned "Italy-ness"
W2 = rng.normal(scale=0.1, size=(2, 8))    # output logits: [Paris, Rome]
W2[0, 0] += 1.5; W2[1, 1] += 1.5

def run(x, patch=None, add=None):
    h = np.maximum(W1 @ x, 0)              # hidden activations
    if patch is not None:                  # overwrite chosen hidden units with stored values
        idx, values = patch
        h = h.copy(); h[idx] = values[idx]
    if add is not None:                    # steering: add a vector to the hidden layer
        h = h + add
    logits = W2 @ h
    return logits[0] - logits[1], h        # logit difference Paris - Rome

clean_x = make_input(FRANCE)
corrupt_x = clean_x.copy(); corrupt_x[:2] = ITALY      # same context, only the country changed
clean_ld, clean_h = run(clean_x)
corrupt_ld, _ = run(corrupt_x)
print(f"clean (France) logit diff {clean_ld:+.2f}, corrupted (Italy) {corrupt_ld:+.2f}")

# Activation patching: run on the corrupted input, but restore one hidden unit from the clean run.
print("\nfraction of the clean behaviour restored by patching each hidden unit:")
for u in range(8):
    patched_ld, _ = run(corrupt_x, patch=([u], clean_h))
    print(f"  unit {u}: {(patched_ld - corrupt_ld) / (clean_ld - corrupt_ld):+.2f}")
both, _ = run(corrupt_x, patch=([0, 1], clean_h))
print(f"  units 0 and 1 together: {(both - corrupt_ld) / (clean_ld - corrupt_ld):+.2f}")

# Steering: difference of mean hidden activations between France and Italy inputs.
h_fr = np.mean([run(make_input(FRANCE))[1] for _ in range(200)], axis=0)
h_it = np.mean([run(make_input(ITALY))[1] for _ in range(200)], axis=0)
steer = h_fr - h_it
print("\nsteering an Italy input towards France:")
for alpha in [0.0, 0.5, 1.0, 1.5]:
    ld, _ = run(corrupt_x, add=alpha * steer)
    print(f"  alpha {alpha:.1f}: logit diff {ld:+.2f} -> answers {'Paris' if ld > 0 else 'Rome'}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Patching localises the behaviour to units 0 and 1, which together restore 98% of it; no single unit does it alone — even in this toy the information is spread over two places, and in real models it usually spreads over many. Notice how the corrupted input was built: identical context, only the country changed. Patching is only as clean as the contrast between the two inputs.

In real models the same idea scales to heads, MLP layers, positions and SAE features, and has been automated: ACDC prunes a model's computational graph to the edges that matter for a task, and gradient-based approximations ("attribution patching") estimate thousands of patches from a couple of backward passes. In 2025 Anthropic introduced attribution graphs, built from cross-layer transcoders, which trace the feature-level computation behind individual outputs of a production model and found, for example, evidence of the model planning a rhyme before writing the line that leads to it (see the Transformer Circuits Thread).

A caution from the editing literature: Meng et al. (2022) used causal tracing to localise factual recall to middle-layer MLPs and edited facts there, but Hase et al. (2023) found that where tracing localises a fact predicts poorly where editing it works best. Causal importance in one experiment does not automatically give a complete or manipulable account.

ExplanationH

Steering: intervening on representations

If concepts are directions, you can push along them. Activation steering adds a vector to the residual stream during generation. The simplest way to get a vector is the difference between mean activations on two contrasting sets of prompts — "love" versus "hate", honest versus dishonest statements — as in ActAdd and Representation Engineering; the toy above did exactly this. SAE features give another source: clamp a feature high or low, as with the Golden Gate Bridge. Researchers have found, for instance, that refusal of harmful requests in several open chat models is largely mediated by a single direction, which can be ablated to remove refusals — a finding with obvious dual-use implications that also says something about how shallow some safety training can be.

Steering is useful as evidence (it shows a direction is causally used), as a lightweight control method, and potentially as a safety tool. Its limits: steering strong enough to change behaviour often degrades fluency or other capabilities; a direction defined by contrast prompts may capture something other than the intended concept; and effects can be inconsistent across prompts. In practice, fine-tuning remains the main way behaviour is shaped; steering is mostly a research instrument as of 2026.

ExplanationH

Honest limits, and where the field stands

It is easy to come away from striking examples — a Golden Gate feature, a rhyme-planning circuit — with an inflated sense of how much we understand. A balanced picture as of 2026:

What works. Linear probes reliably reveal what information is present. Activation patching and its relatives reliably test causal hypotheses about specific behaviours. SAEs and related dictionary methods find large numbers of interpretable, causally relevant features in frontier-scale models. Teams at Anthropic, Google DeepMind and OpenAI, along with academic groups, nonprofits such as EleutherAI, and newer startups, now publish regularly, and open tools and released SAEs let anyone with a laptop study open models. Interpretability has begun to be used in practice: Anthropic, for instance, has reported using interpretability methods in "auditing games" where teams tried to uncover a hidden objective deliberately trained into a model, and in pre-deployment checks.

What doesn't, yet.

  • Coverage. The best analyses explain particular behaviours on particular inputs. Most of what a frontier model computes on a typical input remains unexplained, and SAE reconstruction error means some of it is not even captured by the features.
  • Validation. We lack agreed ways to measure whether an interpretation is correct and complete. Feature labels are written by humans or models looking at examples, and can be wrong in subtle ways ("interpretability illusions"). Some results have not survived closer scrutiny.
  • Scale and cost. Circuit analysis that takes researchers weeks for one behaviour in a small model does not obviously scale to all the behaviours of a model a thousand times bigger. Automation is the main hope.
  • Downstream usefulness. On some practical tasks, simple baselines (a linear probe, a prompt) have matched or beaten sophisticated interpretability methods. The field is increasingly, and healthily, judging methods by whether they help with concrete problems.

Is it on track? Researchers disagree. Optimists see a young science making fast progress that could eventually let us verify properties of models that behaviour alone cannot reveal. Sceptics doubt that a full mechanistic understanding of networks with hundreds of billions of parameters is achievable, and argue for more modest, targeted goals — detecting specific dangerous properties, say — or for relying more on other safety approaches. Both camps agree it is worth doing, and that claims should be backed by causal tests and fair baselines.

Common misconceptionsH

Common misconceptions

Misconception

"Each neuron represents one concept." In language models most neurons are polysemantic, plausibly because of superposition. Interpretable units are usually directions that mix many neurons.

Misconception

"If a probe can decode X, the model uses X." It shows X is present in a decodable form. Use interventions — patching, ablation, steering — to show X affects behaviour.

Misconception

"Attention maps explain the model's decision." They show one ingredient of one head's computation, ignoring values and everything downstream. They suggest hypotheses; they don't confirm them.

Misconception

"SAE features are the model's true, unique features." They are one useful decomposition, which depends on the dictionary size, the sparsity penalty and the training data, and which reconstructs activations imperfectly.

Misconception

"Interpretability is solved" — or — "it is hopeless." Neither. Current tools give real, causally validated insight into specific behaviours and features, and fall far short of a complete account of any frontier model.

Exercises

Exercises

1. Interpretability

CoreComputational

Write a linear probe.

  • fit_probe(X, y): X has shape (n, d) (activations), y is an array of labels in {0, 1}. Fit weights by least squares on targets 2y−12y - 1 (that is, ±1\pm 1), with a bias term: append a column of ones to X and use np.linalg.lstsq. Return the weight vector of length d + 1 (bias last).
  • probe_predict(X, w): return an integer array of 0/1 predictions, 1 where the probe's score is positive.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

2. Interpretability

CoreComputational

Write rank_components(clean, corrupt, patched, threshold). clean and corrupt are the metric values (e.g. logit differences) on the clean and corrupted runs; patched is a list where patched[i] is the metric when component i alone is patched from the clean run into the corrupted run.

Return a list of (i, fraction) pairs, where fraction = (patched[i] - corrupt) / (clean - corrupt), for the components whose fraction is at least threshold, sorted by fraction from largest to smallest. Raise ValueError if clean == corrupt (there is no behaviour to restore).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

3. Interpretability

CoreComputational

Write sae_forward(x, W_enc, b_enc, W_dec, b_dec, lam) for a sparse autoencoder on a batch of activations x of shape (n, d), with W_enc of shape (m, d), b_enc (m,), W_dec (d, m), b_dec (d,).

Compute f=ReLU((x−bdec)Wenc⊤+benc)f = \mathrm{ReLU}((x - b_{\text{dec}}) W_{\text{enc}}^\top + b_{\text{enc}}) and x^=fWdec⊤+bdec\hat x = f W_{\text{dec}}^\top + b_{\text{dec}}. Return (loss, f) where loss is the batch mean of ∥x−x^∥2+λ∥f∥1\lVert x - \hat x \rVert^2 + \lambda \lVert f \rVert_1 (squared error summed over dimensions, L1L_1 summed over latents, then averaged over the batch).

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. Interpretability

Warm-upHuman

A linear probe on layer 18 of a language model predicts whether a sentence is grammatical with 94% held-out accuracy. What can you conclude?

Choose one

5. Interpretability

CoreHuman

Why can a network represent many more features than it has dimensions, as in the lesson's experiment?

Choose one

6. Interpretability

Warm-upHuman

A sparse autoencoder is trained on 512-dimensional activations with 4,096 latents, with separate (untied) encoder and decoder matrices, an encoder bias of size 4,096 and a decoder bias of size 512. How many parameters does it have?

7. Interpretability

CoreHuman

You run a model on a corrupted prompt and patch in one attention head's output from the clean prompt; the correct answer's logit fully recovers. Separately, patching that head's corrupted output into the clean run barely changes the answer. What is the best interpretation?

Choose one

8. Interpretability

StretchComputational

Directional ablation removes a concept direction from activations.

  • mean_difference(acts_a, acts_b): given two arrays of activations of shape (n_a, d) and (n_b, d), return the difference of their means (a minus b), a vector of shape (d,).
  • ablate(X, v): given activations X of shape (n, d) and a non-zero direction v, return X with its component along v removed: x−(x⋅v^)v^x - (x \cdot \hat v)\hat v for each row, where v^=v/∥v∥\hat v = v / \lVert v \rVert.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources