Why build structure into a network?
In Neural networks you built multilayer perceptrons (MLPs): every input connected to every hidden unit. An MLP can in principle approximate any reasonable function. In practice, on raw images, audio or text, it is hopeless, for three reasons.
It is enormous. A modest colour photograph of 224 × 224 pixels has 224 × 224 × 3 = 150,528 numbers. One hidden layer of 1,000 units needs about 150 million weights — before you have learned anything.
It ignores what we know about the data. A cat in the top-left corner of a photo is still a cat in the bottom-right corner. An MLP has to learn that separately for every position, because its weights for one corner have nothing to do with its weights for another.
It assumes a fixed-size input. Sentences have 3 words or 300. An MLP takes exactly as many inputs as it was built for.
The architectures of this lesson each solve one of these problems by building an assumption about the data — an inductive bias — into the network's wiring:
- Convolutional networks assume that useful patterns are local and can appear anywhere. They reuse one small set of weights across every position.
- Recurrent networks assume the data is a sequence. They reuse one set of weights across every time step, carrying a running summary forward.
- Embeddings assume that discrete symbols — words, characters, product ids — have similarities worth learning. They give each symbol a learned vector.
All three survive in today's largest models, in changed forms. And the way the last of them, the recurrent encoder–decoder, broke down is exactly what led to attention and the transformer in the next lesson.
Convolution: one small detector, slid everywhere
Imagine you want to find vertical edges in an image. A reasonable detector looks at a 3 × 3 patch and compares the column on the right with the column on the left. That detector does not care where the patch is. So use the same nine numbers at every position: slide the patch across the image and record the detector's response at each place. That is a convolution.
Definition— Convolution (as used in deep learning)
Given an input of length and a kernel (or filter) of length , the 1-D convolution with stride produces
The 2-D version slides a kernel over an image in the same way. If the input is padded with zeros on each side, the output length is
Mathematicians would call this cross-correlation: a true convolution flips the kernel first. Since the kernel is learned, the flip makes no difference, and deep-learning libraries skip it.
Real layers work on channels. A colour image has 3 input channels; a hidden layer might have 64. A convolutional layer with input channels and output channels has a kernel of shape : each output channel is one detector that looks at every input channel in its window. So the layer has parameters, independent of the image size.
Weight sharing, made concrete
A convolution is a linear map, so it is some matrix. Which one? Each output is a dot product of the same weights with a different window of the input, so the matrix has the kernel on every row, shifted one step to the right each time, and zeros elsewhere. It is a fully connected layer in which almost every weight is forced to be zero (locality) and the rest are forced to be equal along diagonals (weight sharing).
import numpy as np
def conv1d(x, w, stride=1, pad=0):
"""Deep-learning 'convolution' (really cross-correlation) of a 1-D signal."""
x = np.pad(x, pad)
k = len(w)
n_out = (len(x) - k) // stride + 1
return np.array([x[i * stride : i * stride + k] @ w for i in range(n_out)])
def conv2d(img, kernel, stride=1):
"""Single-channel 2-D convolution, 'valid' padding."""
kh, kw = kernel.shape
oh = (img.shape[0] - kh) // stride + 1
ow = (img.shape[1] - kw) // stride + 1
out = np.zeros((oh, ow))
for i in range(oh):
for j in range(ow):
patch = img[i * stride : i * stride + kh, j * stride : j * stride + kw]
out[i, j] = np.sum(patch * kernel)
return out
# 1. A convolution is a matrix multiply with a banded, weight-tied matrix.
x = np.array([1.0, 2.0, 0.0, -1.0, 3.0, 1.0])
w = np.array([0.5, -1.0, 2.0])
n_out = len(x) - len(w) + 1
M = np.zeros((n_out, len(x)))
for i in range(n_out):
M[i, i : i + len(w)] = w # the same three numbers on every row
print("conv1d :", conv1d(x, w))
print("matrix form:", M @ x)
print("matrix has", M.size, "entries but only", len(w), "free parameters")
# 2. Translation equivariance: shift the input, the output shifts too.
signal = np.zeros(12); signal[3] = 1.0; signal[4] = 2.0
shifted = np.roll(signal, 4)
print("\nresponse :", conv1d(signal, w))
print("response, shifted:", conv1d(shifted, w))
# 3. A hand-made vertical-edge detector on a tiny image.
img = np.zeros((6, 6)); img[:, 3:] = 1.0 # dark left half, bright right half
sobel_x = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], dtype=float)
print("\nimage:\n", img)
print("vertical edges:\n", conv2d(img, sobel_x))The second experiment shows the property that makes convolutions right for images: translation equivariance. Shift the input and the output shifts by the same amount, unchanged otherwise. The detector responds to the pattern, not its address. The third experiment uses a classic hand-designed edge filter (the Sobel kernel); a trained network's first layer learns filters much like it on its own.
Seeing more of the image: depth, stride and pooling
One 3 × 3 layer sees a 3 × 3 window. Stack a second 3 × 3 layer on top and each of its outputs depends on a 5 × 5 window of the input; a third gives 7 × 7. This window is the receptive field, and with stride 1 it grows by per layer. To grow it faster, networks downsample: a stride-2 convolution or a pooling layer (taking the maximum or average of each 2 × 2 block) halves the height and width, so later layers see large regions cheaply. The typical convolutional network alternates convolutions and downsampling, trading spatial resolution for more channels as it goes deeper: early layers detect edges and colours, middle layers textures and parts, late layers whole objects.
Example— How much does weight sharing save?
Take a hidden layer of 56 × 56 positions with 64 channels, feeding a layer of 56 × 56 positions with 128 channels.
- Convolution, 3 × 3 kernel: parameters.
- Fully connected: weights.
A factor of about a million. The compute is still substantial — each of the outputs needs multiply-adds, about 231 million in total — but it is the kind of regular, parallel arithmetic that GPUs are built for (see Accelerators and distributed training).
Where convolutional networks went
The modern story starts with LeCun's networks for reading handwritten digits in the 1990s and became a revolution in 2012, when a deep convolutional network trained on GPUs (AlexNet) won the ImageNet image-recognition competition by a wide margin. Networks then grew deeper, until a 2015 change made depth easy: residual connections (He et al.), where each block computes instead of . The identity path lets gradients flow straight through dozens or hundreds of layers. Remember it: every transformer block is built the same way.
Since 2020, vision transformers (Dosovitskiy et al.), which cut an image into patches and treat them like words, have taken over the largest vision and multimodal models. Convolutional networks have not disappeared: a careful modernisation (ConvNeXt) matched transformers at similar compute, convolutions are efficient on phones and embedded devices, and they remain common in audio, in scientific imaging, and as the "stem" that first turns pixels into patches inside many vision transformers.
Recurrent networks: a summary carried through time
Now take a sequence — words, audio frames, daily temperatures. A recurrent neural network (RNN) reads it one element at a time and keeps a hidden state , a fixed-size vector meant to summarise everything so far:
The same , and are used at every step — weight sharing again, this time across time rather than space — so one network handles sequences of any length. An output layer reads to predict, for instance, the next word.
To train it, you unroll the recurrence into a deep feed-forward network with one layer per time step (all sharing weights) and run ordinary backpropagation through it. This is backpropagation through time.
Why the gradients vanish or explode
The loss at step depends on an early state through every step in between. By the chain rule,
That is a product of matrices. Products of many matrices behave like powers: if the factors shrink vectors (largest singular value below 1), the product shrinks exponentially; if they stretch them, it grows exponentially. The derivative is at most 1, so it only ever pushes towards shrinking. Run it:
import numpy as np
rng = np.random.default_rng(0)
d, T = 32, 60
Q, _ = np.linalg.qr(rng.normal(size=(d, d))) # a random orthogonal matrix
xs = rng.normal(scale=0.5, size=(T, d))
def gradient_norms(scale):
"""Norm of d h_t / d h_0 for a tanh RNN whose recurrent matrix is scale * Q."""
W_h = scale * Q
h = np.zeros(d)
J = np.eye(d) # Jacobian d h_t / d h_0, starts as identity
norms = []
for t in range(T):
h = np.tanh(W_h @ h + xs[t])
J = (1 - h**2)[:, None] * (W_h @ J) # chain rule: diag(tanh') @ W_h @ J
norms.append(np.linalg.norm(J, 2)) # largest singular value
return norms
for scale in [0.5, 1.0, 3.0]:
n = gradient_norms(scale)
print(f"scale {scale}: " + " ".join(f"t={t}: {n[t-1]:.1e}" for t in [1, 10, 30, 60]))
# Without the tanh the story is pure linear algebra: the Jacobian is (scale * Q)^t,
# whose norm is exactly scale**t.
for scale in [0.9, 1.1]:
print(f"linear RNN, scale {scale}: after 60 steps the gradient is multiplied by {scale**60:.1e}")With scale 0.5 the signal from step 0 has fallen to after 60 steps — numerically nothing. Even at scale 1.0 the saturation drains it to . At scale 3 it explodes. There is no comfortable middle: the network cannot learn that a word at position 3 matters for a prediction at position 60, because almost no gradient signal connects them. This was analysed in the early 1990s by Hochreiter and by Bengio and colleagues, and revisited by Pascanu, Mikolov and Bengio.
Exploding gradients have a cheap fix: gradient clipping, rescaling the gradient whenever its norm exceeds a threshold. (Large models are still trained with clipping today.) Vanishing gradients need a change of architecture.
Gates: the LSTM idea
The Long Short-Term Memory network (Hochreiter and Schmidhuber, 1997) adds a second state, the cell , updated additively:
where is a candidate update and are gates: sigmoid layers of and that decide, per coordinate, how much of the old cell to keep (forget gate), how much of the new candidate to write (input gate) and how much to expose (output gate). When the forget gate is near 1, : information and gradient pass along the cell unchanged, like the identity path of a residual connection. The GRU (Cho et al., 2014) is a lighter variant with two gates and no separate cell.
Analogy, not a formal construction
A plain RNN is like passing a message along a line of people who each retell it in their own words: after twenty retellings little of the original survives. An LSTM gives the line a notebook that is passed along too. Each person may cross out a few lines or add a few, but whatever nobody touches arrives intact.
LSTMs powered speech recognition, machine translation and handwriting recognition through most of the 2010s. They still had two limits. In practice they remembered tens to a few hundred steps well, not thousands. And — this one matters most for modern hardware — they are sequential: you cannot compute until you have , so a long sequence cannot be processed in parallel during training. Attention removes both limits.
Embeddings: symbols as vectors
Neural networks do arithmetic. Words are not numbers. The simplest encoding gives word number in a vocabulary of size the one-hot vector , all zeros except a 1 in place . But every two one-hot vectors are orthogonal: "tea" is exactly as far from "coffee" as from "volcano". Nothing learned about one word transfers to any other.
Definition— Embedding
An embedding is a learned map from a discrete set of symbols to vectors in , with , stored as a matrix . Symbol is represented by row of . Looking up a row is the same as multiplying the one-hot vector by the matrix, , so an embedding layer is just a linear layer applied to one-hot inputs, implemented as a table lookup.
Similarity between embeddings is usually measured by the angle between them, the cosine similarity
which ignores length (often related to word frequency) and keeps direction.
Where the vectors come from: the distributional hypothesis
The idea, associated with the linguists Zellig Harris and J. R. Firth in the 1950s, is that words that occur in similar contexts tend to have similar meanings. "Tea" and "coffee" both follow "drinks hot" and precede "every morning"; "bus" and "train" both "leave the station". If we describe each word by the company it keeps, similar words get similar descriptions.
The oldest way to do this is to count. Build a matrix of how often each word appears next to each other word, reweight the counts to emphasise surprising pairs (positive pointwise mutual information, PPMI), and compress each long, sparse row into a short dense vector with the singular value decomposition (see Advanced linear algebra):
import numpy as np
corpus = """
amara drinks hot tea every morning . kenji drinks hot coffee every morning .
lucia pours sweet tea for guests . omar pours sweet coffee for guests .
the bus leaves the station at noon . the train leaves the station at dawn .
we took the train across the country . we took the bus across the city .
the cat sleeps on the warm mat . the dog sleeps on the warm rug .
a dog chased the ball in the park . a cat chased the mouse in the garden .
priya drinks cold juice after school . the bus stops near the school .
""".split()
vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)
# 1. Count how often each pair of words appears side by side.
window = 1
C = np.zeros((V, V))
for i, w in enumerate(corpus):
for j in range(max(0, i - window), min(len(corpus), i + window + 1)):
if j != i:
C[idx[w], idx[corpus[j]]] += 1
# 2. Positive pointwise mutual information: how much more often than chance?
total = C.sum()
p_w = C.sum(axis=1, keepdims=True) / total
p_c = C.sum(axis=0, keepdims=True) / total
with np.errstate(divide="ignore"):
pmi = np.log((C / total) / (p_w * p_c))
ppmi = np.maximum(pmi, 0)
# 3. Compress each 1-of-V row to a dense 8-dimensional vector with the SVD.
U, S, _ = np.linalg.svd(ppmi)
E = U[:, :8] * S[:8] # the embedding matrix: one row per word
def cosine(a, b):
return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
def neighbours(word, k=3):
v = E[idx[word]]
sims = [(cosine(v, E[i]), w) for i, w in enumerate(vocab) if w != word]
return sorted(sims, reverse=True)[:k]
for word in ["tea", "train", "cat"]:
print(word, "->", ", ".join(f"{w} ({s:.2f})" for s, w in neighbours(word)))
print(f"\ncos(tea, coffee) = {cosine(E[idx['tea']], E[idx['coffee']]):.2f}")
print(f"cos(tea, train) = {cosine(E[idx['tea']], E[idx['train']]):.2f}")
print("with one-hot vectors both would be exactly 0: every pair of words equally unrelated")Nobody told the program that tea and coffee are both drinks. It found that out from where they appear. With fifteen sentences the other neighbours are noisy; with billions of words the same method gives good vectors.
word2vec: predict, don't count
word2vec (Mikolov and colleagues at Google, 2013) learns the vectors by prediction instead. In its skip-gram form each word has an input vector and an output vector , and training makes large for words actually seen near and small for a handful of randomly drawn "negative" words (negative sampling). It is a logistic regression on word pairs, trained by stochastic gradient descent over a huge corpus — cheap enough to run on billions of words in 2013. Later work showed that skip-gram with negative sampling is closely related to factorising a shifted PMI matrix, so counting and predicting turned out to be two routes to similar vectors. GloVe (Stanford, 2014) made that connection explicit.
The famous party trick is vector arithmetic: the vector for "king" minus "man" plus "woman" lands near "queen". Treat it with care: such analogies work for some relations and not others, and the standard evaluation quietly excludes the input words from the answers, which flatters the results. More importantly, embeddings learn all the regularities of their text, including stereotypes: Bolukbasi et al. (2016) found occupational gender bias in widely used word vectors. A model is a mirror of its data before it is anything else.
Embeddings today
Every language model in this course starts with an embedding table: the first layer maps each token id to a vector of a few thousand dimensions. Two things changed after word2vec. First, the units are now subword tokens rather than words (Language models). Second, the vector from the table is only the starting point: each layer of a transformer updates it using the surrounding context, so "bank" in "river bank" and in "bank loan" ends up with different contextual representations. Separately trained embedding models, which map a whole sentence or document to one vector, are the backbone of semantic search and retrieval-augmented generation (Reasoning models, tools and agents).
Sequence to sequence, and the bottleneck
Machine translation maps a sequence to another sequence of a different length. In 2014 Sutskever, Vinyals and Le and Cho et al. proposed the encoder–decoder (or seq2seq) design:
- An encoder RNN reads the source sentence, word by word, and its final hidden state becomes the context vector .
- A decoder RNN starts from and generates the target sentence one word at a time, each step feeding back the word it just produced, until it emits an end-of-sentence token.
Trained end to end on pairs of sentences, it worked remarkably well, and within two years neural translation systems had replaced the older phrase-based ones in production.
But look at step 1. Everything the decoder will ever know about the source sentence must pass through one fixed-size vector — say 1,000 numbers — whether the sentence has five words or fifty. That is the bottleneck. Researchers measured the consequence directly: translation quality dropped as sentences got longer. Reversing the source sentence, a trick Sutskever and colleagues found helpful, eased it by putting the first source words close to the first target words; it did not remove it.
Bahdanau, Cho and Bengio (2014) proposed the fix. Keep all the encoder's hidden states , not just the last. When the decoder produces target word , let it compute a relevance score between its own state and each , turn the scores into weights with a softmax, and read a weighted average of the — a different context vector for every output word. Translating "the black cat" into Swahili or Spanish, the decoder can look mostly at "cat" when it writes the noun and at "black" when it writes the adjective, wherever they sit in the source. The model learned these soft alignments by itself.
They called it attention. Three years later a team at Google asked what happens if you keep the attention and throw away the recurrence entirely. That is the next lesson.
Common misconceptions
Misconception
"A convolutional layer flips its kernel, as in signal processing." Deep-learning libraries compute cross-correlation, without the flip. Because the kernel is learned, the two are equivalent; the distinction matters only when you load hand-designed kernels or compare with a signal-processing textbook.
Misconception
"CNNs are translation invariant." Convolutions are translation equivariant: shifting the input shifts the feature map. Invariance (the output not changing at all) comes only approximately, from pooling and the final global average, and real networks are measurably sensitive to small shifts because strided downsampling breaks exact equivariance.
Misconception
"LSTMs solved long-range dependencies." Gating greatly eased the vanishing gradient, but LSTMs still struggled beyond a few hundred steps and could not be parallelised over time. Both facts motivated attention.
Misconception
"Recurrent networks are obsolete." Pure RNNs are rare in large language models, but recurrence has returned in new forms. State-space models such as Mamba and related linear-attention layers keep a fixed-size state, run in time linear in sequence length, and can be trained in parallel. Several labs have released hybrid models that interleave a few attention layers with many of these recurrent layers; whether hybrids will displace pure transformers at the frontier is an open, actively tested question as of 2026.
Misconception
"Word embeddings capture meaning." They capture distributional statistics — which words keep company with which — and that correlates with meaning surprisingly well. It also captures the biases, gaps and quirks of the corpus, and says nothing about words that are rare in it.
Exercises
1. Convolutions, sequences and embeddings
Warm-upHumanA 1-D convolution has input length , kernel size , stride and padding on each side. What is the output length?
2. Convolutions, sequences and embeddings
Warm-upHumanA 2-D convolutional layer has a kernel, 32 input channels and 64 output channels, with one bias per output channel. How many parameters does it have?
3. Convolutions, sequences and embeddings
CoreHumanFour convolutional layers with stride 1 and no pooling are stacked. One output position of the fourth layer depends on a square window of the original input. What is the side length of that window?
4. Convolutions, sequences and embeddings
CoreComputationalWrite conv2d_multi(x, w, b) for a full convolutional layer (stride 1, no padding).
xhas shape(C_in, H, W).whas shape(C_out, C_in, kh, kw).bhas shape(C_out,).- Return an array of shape
(C_out, H - kh + 1, W - kw + 1)where output channeloat position(i, j)isb[o]plus the sum over all input channels and kernel positions ofw[o, c, u, v] * x[c, i + u, j + v].
Use numpy; loops over output positions are fine.
5. Convolutions, sequences and embeddings
CoreComputationalWrite rnn_forward(xs, W_x, W_h, b) for a vanilla RNN.
xshas shape(T, n_in): one input vector per time step.W_xhas shape(d, n_in),W_hhas shape(d, d),bhas shape(d,).- Start from and apply for .
- Return an array of shape
(T, d)whose row is .
6. Convolutions, sequences and embeddings
CoreComputationalWrite most_similar(E, vocab, word, k).
Eis an embedding matrix of shape(V, d); rowiis the vector ofvocab[i].- Return a list of the
kwords (strings) with the highest cosine similarity toword, most similar first, excludingworditself.
7. Convolutions, sequences and embeddings
CoreHumanWhy does a vanilla RNN struggle to learn that an input 100 steps ago affects the current output?
8. Convolutions, sequences and embeddings
CoreHumanIn a 2014-style encoder–decoder translation model, what exactly is the bottleneck, and what did attention change?
Further reading and resources
- Deep Residual Learning for Image Recognition · He, Zhang, Ren, Sun · 2015Residual connections, which made very deep networks trainable and reappear in every transformer.
- A ConvNet for the 2020s · Liu et al. · 2022A careful modernisation of convolutional networks that holds its own against vision transformers.
- Long Short-Term Memory · Hochreiter, Schmidhuber · 1997The original LSTM paper.
- On the difficulty of training recurrent neural networks · Pascanu, Mikolov, Bengio · 2012The vanishing and exploding gradient problem analysed, and gradient clipping proposed.
- Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation · Cho et al. · 2014Introduces the encoder–decoder RNN and the gated recurrent unit (GRU).
- Efficient Estimation of Word Representations in Vector Space · Mikolov, Chen, Corrado, Dean · 2013The word2vec paper.
- Distributed Representations of Words and Phrases and their Compositionality · Mikolov et al. · 2013Skip-gram with negative sampling.
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings · Bolukbasi et al. · 2016Embeddings learn the statistics of their text, including its social biases.
- Sequence to Sequence Learning with Neural Networks · Sutskever, Vinyals, Le · 2014
- Neural Machine Translation by Jointly Learning to Align and Translate · Bahdanau, Cho, Bengio · 2014The paper that introduced attention to fix the encoder–decoder bottleneck.
- Mamba: Linear-Time Sequence Modeling with Selective State Spaces · Gu, Dao · 2023A modern recurrent (state-space) architecture.
- CS231n: Deep Learning for Computer Vision — course notesStanford's free notes on convolutional networks.
- Understanding LSTM Networks · Christopher Olah · 2015A clear illustrated walk through the LSTM's gates.