Keep the attention, drop the recurrence
The previous lesson ended with two problems and a hint. The problems: a recurrent encoder squeezes a sentence into one vector, and recurrent networks process tokens one after another, so they cannot use a GPU's parallelism across a sequence. The hint: Bahdanau's attention let a decoder look back at every encoder state directly, with learned weights.
In 2017, eight researchers at Google published "Attention Is All You Need". Their transformer threw away recurrence and convolution and built a translation model from attention alone, plus small per-position MLPs. Every token looks at every other token in one step, so there is no long chain for gradients to vanish along, and every position of a training sequence is computed in parallel. It beat the best translation systems of the day at a fraction of their training cost.
Nearly a decade later the core block is little changed, and it is the architecture inside essentially every frontier language model, most large vision and speech models, protein-structure predictors and much more. This lesson builds it from the ground up in numpy, then counts what it costs. Everything in the Frontier level of this course assumes you know this lesson well.
Attention as a soft dictionary lookup
A Python dictionary maps keys to values. You present a query; if it equals a key exactly, you get that key's value. Attention softens every part of this:
- the query is compared with every key, by a similarity score (a dot product), not tested for equality;
- the scores are turned into weights that are positive and sum to 1 (a softmax);
- the result is the weighted average of all the values.
A query that matches one key much better than the others gets, in effect, that key's value. A vague query gets a blend. And because every step is differentiable, the network can learn what to put in queries, keys and values.
Analogy, not a formal construction
Think of a librarian helping a reader. The reader's question is the query; each book's catalogue entry is a key; the book's contents are the value. A good librarian does not hand over one book: they pull the few that match best, and the reader takes most from the best match and a little from the others. In self-attention every word in the sentence is at once a reader with a question and a book on the shelf.
Scaled dot-product attention
Definition— Scaled dot-product attention
Given queries , keys and values ,
where the softmax is applied to each row. Row of the matrix of attention weights says how much query draws on each value; it is a probability distribution.
In self-attention the queries, keys and values all come from the same sequence , through learned projections: , , .
Why divide by ? If the entries of a query and a key are independent with mean 0 and variance 1, their dot product is a sum of such products and has variance . With the raw scores would have a standard deviation around 11, the softmax would put almost all weight on the single largest score, and its gradient with respect to the others would be nearly zero. Dividing by brings the scores back to unit scale. It is a small detail that the paper called out, and it matters.
import numpy as np
def softmax(z, axis=-1):
z = z - z.max(axis=axis, keepdims=True) # subtracting the max changes nothing but avoids overflow
e = np.exp(z)
return e / e.sum(axis=axis, keepdims=True)
def attention(Q, K, V, mask=None):
"""Scaled dot-product attention. Q: (n, d_k), K: (m, d_k), V: (m, d_v)."""
scores = Q @ K.T / np.sqrt(K.shape[-1]) # (n, m): how well each query matches each key
if mask is not None:
scores = np.where(mask, scores, -np.inf)
weights = softmax(scores) # each row is a probability distribution
return weights @ V, weights
# A soft dictionary: three keys, each with a value.
K = np.array([[2.0, 0.0], [0.0, 2.0], [-1.0, -1.0]])
V = np.array([[10.0, 0.0], [0.0, 10.0], [5.0, 5.0]])
Q = np.array([[3.0, 0.0], # a query that strongly matches key 0
[0.2, 0.2]]) # a vague query
out, w = attention(Q, K, V)
np.set_printoptions(precision=3, suppress=True)
print("weights:\n", w)
print("row sums:", w.sum(axis=1))
print("outputs:\n", out)
# Self-attention has no idea of order: permute the inputs and the outputs permute the same way.
rng = np.random.default_rng(0)
n, d = 5, 8
X = rng.normal(size=(n, d))
Wq, Wk, Wv = (rng.normal(size=(d, d)) for _ in range(3))
Y, _ = attention(X @ Wq, X @ Wk, X @ Wv)
perm = np.array([3, 0, 4, 1, 2])
Yp, _ = attention(X[perm] @ Wq, X[perm] @ Wk, X[perm] @ Wv)
print("\npermutation equivariant:", np.allclose(Yp, Y[perm]))
# Why divide by sqrt(d_k): dot products of random unit-variance vectors have variance d_k.
for d_k in [4, 64, 1024]:
q = rng.normal(size=(10000, d_k)); k = rng.normal(size=(10000, d_k))
dots = (q * k).sum(axis=1)
print(f"d_k={d_k:5d}: std of q.k = {dots.std():6.1f}, after scaling = {(dots / np.sqrt(d_k)).std():.2f}")Three things to take from the output. The sharp query takes 98% of its answer from key 0, the vague one blends all three. Every row of weights sums to 1. And self-attention is permutation equivariant: shuffle the tokens and the outputs are shuffled in the same way, otherwise identical. Attention by itself treats a sentence as a bag of words. "Dog bites man" and "man bites dog" look the same to it. We will fix that with positional information below.
Cost. The score matrix has entries for a sequence of tokens, so self-attention takes time and, naively, memory. That quadratic term is the transformer's most famous weakness; Accelerators and distributed training shows how FlashAttention removes the memory part of it, though not the time.
Multiple heads and the causal mask
One attention operation computes one weighted average per position, so it can follow one kind of relationship at a time. Language has many at once: which noun an adjective modifies, what a pronoun refers to, what the previous token was. Multi-head attention runs attentions in parallel on smaller projections and concatenates the results:
Each head works in dimensions, so heads cost the same as one full-width head. In code you compute once at full width and reshape it into slices.
The causal mask
A language model is trained to predict token from tokens (next lesson). During training we feed it the whole sequence at once and ask for a prediction at every position, so we must stop position from peeking at later positions. The fix is a causal mask: set the scores for keys after the query to before the softmax, so their weights become exactly 0. The weight matrix becomes lower-triangular.
import numpy as np
def softmax(z, axis=-1):
z = z - z.max(axis=axis, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=axis, keepdims=True)
def multi_head_attention(X, Wq, Wk, Wv, Wo, n_heads, causal=True):
"""X: (n, d_model). Each W: (d_model, d_model). Returns (n, d_model) and the weights."""
n, d_model = X.shape
d_head = d_model // n_heads
# Project, then split the last dimension into heads: (n_heads, n, d_head).
def split(M):
return M.reshape(n, n_heads, d_head).transpose(1, 0, 2)
Q, K, V = split(X @ Wq), split(X @ Wk), split(X @ Wv)
scores = Q @ K.transpose(0, 2, 1) / np.sqrt(d_head) # (n_heads, n, n)
if causal:
future = np.triu(np.ones((n, n), dtype=bool), k=1) # True above the diagonal
scores = np.where(future, -np.inf, scores)
weights = softmax(scores)
heads = weights @ V # (n_heads, n, d_head)
concat = heads.transpose(1, 0, 2).reshape(n, d_model) # glue the heads back together
return concat @ Wo, weights
rng = np.random.default_rng(1)
n, d_model, n_heads = 6, 16, 4
Wq, Wk, Wv, Wo = (rng.normal(scale=d_model**-0.5, size=(d_model, d_model)) for _ in range(4))
X = rng.normal(size=(n, d_model))
Y, W = multi_head_attention(X, Wq, Wk, Wv, Wo, n_heads)
np.set_printoptions(precision=2, suppress=True)
print("head 0 attention weights (row = query position):\n", W[0])
# Causality: change the last two tokens; outputs at positions 0..3 must not move.
X2 = X.copy()
X2[4:] = rng.normal(size=(2, d_model))
Y2, _ = multi_head_attention(X2, Wq, Wk, Wv, Wo, n_heads)
print("positions 0-3 unchanged:", np.allclose(Y[:4], Y2[:4]))
print("positions 4-5 changed: ", not np.allclose(Y[4:], Y2[4:]))The causality test is worth copying into any transformer you write: perturb the future, and check that the past does not move. A leak here produces a model that looks brilliant during training (it can read the answer) and is useless at generation time.
Heads in trained models do specialise. Interpretability researchers have found heads that attend to the previous token, heads that find an earlier occurrence of the current token and copy what followed it (induction heads), and heads that track syntax. Interpretability returns to them, and to why attention weights alone are not an explanation.
Fewer keys and values: MQA and GQA
At generation time the keys and values of past tokens are cached, and that cache can be larger than the model itself (Serving models). Multi-query attention (Shazeer, 2019) lets all heads share one key and one value head; grouped-query attention (Ainslie et al., 2023) is the compromise most open models use today: say 32 query heads sharing 8 key–value heads. The quality cost is small and the memory saving large.
Where am I? Positional information
Since attention is permutation equivariant, order has to be supplied from outside. There are three main families.
Absolute encodings, added to the input. The original transformer added a fixed vector to each token's embedding:
Each pair of dimensions is a clock hand turning at its own speed, from once every positions to once every . Together they give each position a distinct fingerprint, and a fixed offset corresponds to a fixed rotation of every pair (Fourier and harmonic analysis). GPT-2 and BERT instead learned a vector per position, which works well but cannot go past the longest position seen in training.
Relative position in the scores. What usually matters is how far apart two tokens are, not their absolute addresses. ALiBi (Press et al.) simply subtracts a penalty proportional to the distance from each attention score.
Rotary position embedding (RoPE) (Su et al.) is what most current open models use. It does not add anything to the embeddings. Instead, just before the dot product, it rotates each pair of coordinates of a query at position by the angle , and each pair of a key at position by . Rotating two 2-D vectors by angles and changes their dot product exactly as rotating one of them by would. So the score between a query and a key depends on their content and on only.
import numpy as np
def sinusoidal(n_pos, d):
"""The original transformer's fixed positional encodings, shape (n_pos, d)."""
pos = np.arange(n_pos)[:, None]
freq = 10000.0 ** (-np.arange(0, d, 2) / d) # one frequency per pair of dimensions
pe = np.zeros((n_pos, d))
pe[:, 0::2] = np.sin(pos * freq)
pe[:, 1::2] = np.cos(pos * freq)
return pe
def rope(x, pos, base=10000.0):
"""Rotary position embedding: rotate each pair (x[2i], x[2i+1]) by the angle pos * theta_i."""
d = x.shape[-1]
theta = base ** (-np.arange(0, d, 2) / d)
angle = pos * theta
cos, sin = np.cos(angle), np.sin(angle)
x_even, x_odd = x[0::2], x[1::2]
out = np.empty_like(x)
out[0::2] = x_even * cos - x_odd * sin
out[1::2] = x_even * sin + x_odd * cos
return out
pe = sinusoidal(50, 8)
print("sinusoidal encodings of positions 0-2 (first 4 dims):\n", np.round(pe[:3, :4], 3))
rng = np.random.default_rng(0)
q, k = rng.normal(size=64), rng.normal(size=64)
# The score between a query at position m and a key at position n depends only on m - n.
for m, n in [(5, 2), (105, 102), (1005, 1002)]:
print(f"q at {m:4d}, k at {n:4d} (offset 3): score = {rope(q, m) @ rope(k, n):.6f}")
print(f"q at 5, k at 1 (offset 4): score = {rope(q, 5) @ rope(k, 1):.6f}")
# Rotations preserve length, so RoPE never changes the size of a vector.
print("norm before/after:", round(float(np.linalg.norm(q)), 6), round(float(np.linalg.norm(rope(q, 77))), 6))The same query–key pair scores identically at positions (5, 2), (105, 102) and (1005, 1002), and differently at offset 4. That relative property, plus the fact that rotations do not distort vector lengths, is why RoPE became the default. It also made context extension possible: by rescaling the rotation frequencies (position interpolation, and refinements such as YaRN), labs stretch a model trained on, say, 8,000-token sequences to work on 128,000 tokens or more with a short additional training phase. How well models actually use very long contexts is a separate question, taken up in Evaluating models.
The transformer block
Attention moves information between positions. The block also needs computation within each position, and the machinery that makes a deep stack trainable.
Definition— Transformer block (pre-norm)
For a sequence of vectors , one block computes
where the MLP is applied to each position independently, classically with and , and Norm is layer normalisation: subtract each vector's mean, divide by its standard deviation, then apply a learned gain and bias. A model is an embedding layer, blocks, a final norm, and an output ("unembedding") layer that maps each position's vector to a score for every token in the vocabulary.
Each ingredient has a job:
- Residual connections (from ResNets, previous lesson) give gradients an identity path through all blocks, which is what lets 100-layer transformers train at all. They also suggest a useful picture: a residual stream of vectors flowing through the model, which each sublayer reads from and writes into by addition (Elhage et al.). We will lean on this picture in Interpretability.
- Normalisation keeps activations at a stable scale. The 2017 paper normalised after adding the residual ("post-norm"); almost everyone now normalises the input of each sublayer ("pre-norm"), which trains more stably in deep stacks. Many models replace LayerNorm with the cheaper RMSNorm, which skips subtracting the mean.
- The MLP holds about two-thirds of the parameters and is where much of a model's stored knowledge appears to live. Current open models mostly use a gated variant, SwiGLU (Shazeer, 2020): , with the hidden width reduced to about so the parameter count stays the same.
Here is a whole, if tiny, decoder-only language model: embeddings, two pre-norm blocks, and an output layer that reuses the embedding matrix (weight tying, common in smaller models).
import numpy as np
def softmax(z):
z = z - z.max(axis=-1, keepdims=True)
e = np.exp(z)
return e / e.sum(axis=-1, keepdims=True)
def layer_norm(x, gain, bias, eps=1e-5):
mu = x.mean(axis=-1, keepdims=True)
var = x.var(axis=-1, keepdims=True)
return gain * (x - mu) / np.sqrt(var + eps) + bias
def gelu(x):
return 0.5 * x * (1 + np.tanh(np.sqrt(2 / np.pi) * (x + 0.044715 * x**3)))
def init_block(rng, d, ff_mult=4):
s = d ** -0.5
return {
"ln1_g": np.ones(d), "ln1_b": np.zeros(d),
"Wq": rng.normal(scale=s, size=(d, d)), "Wk": rng.normal(scale=s, size=(d, d)),
"Wv": rng.normal(scale=s, size=(d, d)), "Wo": rng.normal(scale=s, size=(d, d)),
"ln2_g": np.ones(d), "ln2_b": np.zeros(d),
"W1": rng.normal(scale=s, size=(d, ff_mult * d)), "b1": np.zeros(ff_mult * d),
"W2": rng.normal(scale=(ff_mult * d) ** -0.5, size=(ff_mult * d, d)), "b2": np.zeros(d),
}
def block(x, p, n_heads):
"""One pre-norm decoder block: x + Attn(LN(x)), then x + MLP(LN(x))."""
n, d = x.shape
dh = d // n_heads
h = layer_norm(x, p["ln1_g"], p["ln1_b"])
split = lambda M: M.reshape(n, n_heads, dh).transpose(1, 0, 2)
Q, K, V = split(h @ p["Wq"]), split(h @ p["Wk"]), split(h @ p["Wv"])
scores = Q @ K.transpose(0, 2, 1) / np.sqrt(dh)
scores = np.where(np.triu(np.ones((n, n), dtype=bool), 1), -np.inf, scores)
attn = (softmax(scores) @ V).transpose(1, 0, 2).reshape(n, d) @ p["Wo"]
x = x + attn # residual connection 1
h = layer_norm(x, p["ln2_g"], p["ln2_b"])
x = x + gelu(h @ p["W1"] + p["b1"]) @ p["W2"] + p["b2"] # residual connection 2
return x
# A whole (tiny) decoder-only language model.
rng = np.random.default_rng(0)
V_size, d, L, n_heads = 100, 32, 2, 4
emb = rng.normal(scale=0.02, size=(V_size, d)) # token embeddings, shared with the output layer
pos = rng.normal(scale=0.02, size=(64, d)) # learned absolute positions
blocks = [init_block(rng, d) for _ in range(L)]
lnf_g, lnf_b = np.ones(d), np.zeros(d)
tokens = np.array([5, 17, 42, 42, 7])
x = emb[tokens] + pos[: len(tokens)]
for p in blocks:
x = block(x, p, n_heads)
logits = layer_norm(x, lnf_g, lnf_b) @ emb.T # (n, V): a score for every next token
print("logits shape:", logits.shape)
print("next-token distribution after the last position sums to", round(float(softmax(logits[-1]).sum()), 6))
# Count parameters and compare with the rule of thumb 12 * L * d^2.
n_block = sum(a.size for a in blocks[0].values())
total = n_block * L + emb.size + pos.size + 2 * d
print(f"\nper block: {n_block} (12 d^2 = {12 * d * d}; the rest are biases and norm gains)")
print(f"total: {total} of which embeddings: {emb.size + pos.size}")The model is untrained, so its predictions are noise; training it is the subject of the next lesson. But every piece of a GPT-style model is here, and production code differs mainly in scale, in fused GPU kernels, and in the refinements listed above.
As of 2026 a typical open-weight decoder (the Llama, Qwen, Mistral, DeepSeek, Gemma and OLMo families, among others) uses pre-norm RMSNorm, RoPE, SwiGLU, grouped-query attention, no bias terms, and increasingly a mixture-of-experts MLP. Individual families add their own twists — normalising queries and keys, alternating full attention with sliding-window layers, compressing the key–value cache — and closed frontier labs publish fewer details, but nothing suggests their cores are fundamentally different.
Encoders, decoders and both
The original transformer was an encoder–decoder for translation. Three variants descend from it:
| Variant | Attention | Trained to | Examples | Good for |
|---|---|---|---|---|
| Encoder-only | bidirectional: every token sees every token | fill in masked-out tokens | BERT and its descendants | classification, search embeddings, tagging |
| Decoder-only | causal: each token sees only earlier ones | predict the next token | GPT series and almost every current LLM | generation of any kind |
| Encoder–decoder | encoder bidirectional; decoder causal, plus cross-attention | map an input sequence to an output sequence | original transformer, T5, many speech models | translation, transcription, summarisation |
In cross-attention the queries come from the decoder and the keys and values from the encoder's output — exactly Bahdanau's 2014 mechanism, now without the RNN.
Why did decoder-only models win the language-model race? One objective — predict the next token — turns every text on the internet into training data, scales smoothly, and yields a model that can do any task phrased as text continuation, translation included. Encoder–decoder models remain competitive where input and output are clearly separate (speech recognition is a common case), and bidirectional encoders still power much of search and classification because they are small and fast.
Counting parameters and FLOPs
You should be able to estimate a transformer's size and training cost on the back of an envelope. Ignore biases and norms (tiny), and use for the model width, for the number of blocks, for the vocabulary size.
Parameters per block. Attention has four matrices (): . The MLP has and : . Total per block, so
the second term being the embedding table (doubled if the output layer is not tied to it).
FLOPs. A matrix–vector product with an matrix costs multiply–adds, that is floating-point operations. Every weight is used once per token in the forward pass, so the forward pass costs about FLOPs per token. The backward pass computes gradients with respect to both activations and weights and costs about twice the forward. Hence the rule used throughout the field:
The attention scores add a term the parameter count does not see: for each token and layer, the dot products with keys and the weighted sum of values cost about FLOPs. With a causal mask a token attends to positions on average, so about . Relative to the FLOPs per token and layer spent on the weights, that is a fraction of about . For typical widths and contexts of a few thousand tokens it is small; for contexts of hundreds of thousands of tokens it dominates.
Example— GPT-3 on the back of an envelope
GPT-3's largest model (Brown et al., 2020) has layers, and a vocabulary of about 50,000 tokens.
- Blocks: .
- Embeddings: .
- Total: about 175 billion — the published figure.
It was trained on about 300 billion tokens, so FLOPs, which matches the paper's own estimate. With a context of 2,048 tokens, attention scores add only about .
These rules are how labs budget training runs, and you will use them in every remaining lesson: to pick model sizes (Language models), to plan clusters (Accelerators) and to price inference (Serving models).
Common misconceptions
Misconception
"The attention weights show what the model is thinking about." They show how one head mixes value vectors in one layer. The values themselves can be large or near zero, many heads and layers combine, and quite different weight patterns can produce similar outputs. Attention maps are a useful hint, not an explanation (Interpretability).
Misconception
"Transformers understand word order because they read left to right." Attention itself is order-blind; order enters only through positional information. Even a causal mask gives only a weak, implicit sense of position. Remove the positional scheme from a bidirectional model and it really does see a bag of words.
Misconception
"The quadratic cost of attention dominates everything." For contexts of a few thousand tokens the MLPs and projections, which grow with , cost far more. The quadratic term takes over only at long contexts, which is exactly where FlashAttention, sparse and sliding-window attention, and recurrent hybrids come in.
Misconception
"The 4× MLP width, 8 heads and 10,000 RoPE base are laws." They are conventions that worked. Production models vary all of them; the RoPE base, for instance, is routinely raised to support longer contexts.
Misconception
"Decoder-only models can't do understanding tasks; you need BERT for that." A large decoder-only model, prompted or fine-tuned, does classification and extraction very well. Encoders persist because they are cheap, not because decoders can't.
Exercises
1. Attention and the transformer
Warm-upHumanA transformer block has width , four attention matrices and an MLP with hidden width . Ignoring biases and normalisation parameters, how many parameters does one block have?
2. Attention and the transformer
Warm-upHumanEstimate the training compute of a 1-billion-parameter transformer trained on 20 billion tokens, using . Give the answer in units of FLOPs (an exaFLOP).
3. Attention and the transformer
CoreHumanWhy does scaled dot-product attention divide the scores by ?
4. Attention and the transformer
CoreComputationalWrite causal_attention(Q, K, V) for a single head. Q and K have shape (n, d_k), V has shape (n, d_v). Return the (n, d_v) output of scaled dot-product attention in which position i may attend only to positions 0, ..., i.
Make your softmax numerically stable (subtract the row maximum).
5. Attention and the transformer
CoreComputationalWrite sinusoidal(n_pos, d) returning the original transformer's positional encodings as an array of shape (n_pos, d) (assume d is even):
and .
6. Attention and the transformer
CoreHumanWhich statements are true? Select all that apply.
7. Attention and the transformer
StretchComputationalWrite count_params(V, d, L, ff_mult, n_ctx) returning the exact number of parameters of this decoder-only model:
- token embedding table
V × d, tied with the output layer (counted once); - learned position embeddings
n_ctx × d; Lblocks, each with: four attention matricesd × d(no biases); an MLP withW1of shaped × (ff_mult·d)plus bias, andW2of shape(ff_mult·d) × dplus bias; two LayerNorms, each with a gain and a bias of sized;- a final LayerNorm (gain and bias).
The tiny model in the lesson (V=100, d=32, L=2, ff_mult=4, n_ctx=64) has 30,464 parameters.
8. Attention and the transformer
StretchHumanIn a causal decoder of width , the attention scores cost about FLOPs per token per layer and the weight matrices about . At what context length are the two equal?
Further reading and resources
- Attention Is All You Need · Vaswani et al. · 2017The transformer paper.
- Neural Machine Translation by Jointly Learning to Align and Translate · Bahdanau, Cho, Bengio · 2014
- Layer Normalization · Ba, Kiros, Hinton · 2016
- On Layer Normalization in the Transformer Architecture · Xiong et al. · 2020Why pre-norm blocks train more stably than the original post-norm ones.
- Root Mean Square Layer Normalization · Zhang, Sennrich · 2019
- GLU Variants Improve Transformer · Noam Shazeer · 2020The SwiGLU feed-forward layer used in most current open models.
- RoFormer: Enhanced Transformer with Rotary Position Embedding · Su et al. · 2021
- Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation · Press, Smith, Lewis · 2021
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Devlin et al. · 2018
- Language Models are Few-Shot Learners · Brown et al. · 2020GPT-3, whose sizes are used in this lesson's worked example.
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer · Raffel et al. · 2019T5, an encoder–decoder model.
- Fast Transformer Decoding: One Write-Head is All You Need · Noam Shazeer · 2019Multi-query attention.
- GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints · Ainslie et al. · 2023
- Scaling Laws for Neural Language Models · Kaplan et al. · 2020Its Table 1 is the standard accounting of transformer FLOPs.
- A Mathematical Framework for Transformer Circuits · Elhage et al. (Anthropic) · 2021The residual-stream view of the transformer.
- nanoGPT · Andrej KarpathyA small, readable GPT implementation in PyTorch.