Skip to content
LumenaryAcademy
ML 65 min

Convolutions, sequences and embeddings

Three ways to build structure into a network — share weights across space, carry state through time, turn symbols into vectors — and the bottleneck that led to attention.

Code tested · 6AI-draftedAwaiting review by a computer scientist

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Why build structure into a network?

In Neural networks you built multilayer perceptrons (MLPs): every input connected to every hidden unit. An MLP can in principle approximate any reasonable function. In practice, on raw images, audio or text, it is hopeless, for three reasons.

It is enormous. A modest colour photograph of 224 × 224 pixels has 224 × 224 × 3 = 150,528 numbers. One hidden layer of 1,000 units needs about 150 million weights — before you have learned anything.

It ignores what we know about the data. A cat in the top-left corner of a photo is still a cat in the bottom-right corner. An MLP has to learn that separately for every position, because its weights for one corner have nothing to do with its weights for another.

It assumes a fixed-size input. Sentences have 3 words or 300. An MLP takes exactly as many inputs as it was built for.

The architectures of this lesson each solve one of these problems by building an assumption about the data — an inductive bias — into the network's wiring:

  • Convolutional networks assume that useful patterns are local and can appear anywhere. They reuse one small set of weights across every position.
  • Recurrent networks assume the data is a sequence. They reuse one set of weights across every time step, carrying a running summary forward.
  • Embeddings assume that discrete symbols — words, characters, product ids — have similarities worth learning. They give each symbol a learned vector.

All three survive in today's largest models, in changed forms. And the way the last of them, the recurrent encoder–decoder, broke down is exactly what led to attention and the transformer in the next lesson.

DefinitionsH

Convolution: one small detector, slid everywhere

Imagine you want to find vertical edges in an image. A reasonable detector looks at a 3 × 3 patch and compares the column on the right with the column on the left. That detector does not care where the patch is. So use the same nine numbers at every position: slide the patch across the image and record the detector's response at each place. That is a convolution.

Definition— Convolution (as used in deep learning)

Given an input xx of length nn and a kernel (or filter) ww of length kk, the 1-D convolution with stride ss produces

yi=∑j=0k−1wj xis+j+b,i=0,1,…,⌊n−ks⌋.y_i = \sum_{j=0}^{k-1} w_j \, x_{i s + j} + b, \qquad i = 0, 1, \ldots, \left\lfloor \frac{n - k}{s} \right\rfloor.

The 2-D version slides a kh×kwk_h \times k_w kernel over an image in the same way. If the input is padded with pp zeros on each side, the output length is

nout=⌊n+2p−ks⌋+1.n_{\text{out}} = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1 .

Mathematicians would call this cross-correlation: a true convolution flips the kernel first. Since the kernel is learned, the flip makes no difference, and deep-learning libraries skip it.

Real layers work on channels. A colour image has 3 input channels; a hidden layer might have 64. A convolutional layer with CinC_{\text{in}} input channels and CoutC_{\text{out}} output channels has a kernel of shape Cout×Cin×kh×kwC_{\text{out}} \times C_{\text{in}} \times k_h \times k_w: each output channel is one detector that looks at every input channel in its window. So the layer has CoutCinkhkw+CoutC_{\text{out}} C_{\text{in}} k_h k_w + C_{\text{out}} parameters, independent of the image size.

Weight sharing, made concrete

A convolution is a linear map, so it is some matrix. Which one? Each output is a dot product of the same kk weights with a different window of the input, so the matrix has the kernel on every row, shifted one step to the right each time, and zeros elsewhere. It is a fully connected layer in which almost every weight is forced to be zero (locality) and the rest are forced to be equal along diagonals (weight sharing).

python
import numpy as np

def conv1d(x, w, stride=1, pad=0):
    """Deep-learning 'convolution' (really cross-correlation) of a 1-D signal."""
    x = np.pad(x, pad)
    k = len(w)
    n_out = (len(x) - k) // stride + 1
    return np.array([x[i * stride : i * stride + k] @ w for i in range(n_out)])

def conv2d(img, kernel, stride=1):
    """Single-channel 2-D convolution, 'valid' padding."""
    kh, kw = kernel.shape
    oh = (img.shape[0] - kh) // stride + 1
    ow = (img.shape[1] - kw) // stride + 1
    out = np.zeros((oh, ow))
    for i in range(oh):
        for j in range(ow):
            patch = img[i * stride : i * stride + kh, j * stride : j * stride + kw]
            out[i, j] = np.sum(patch * kernel)
    return out

# 1. A convolution is a matrix multiply with a banded, weight-tied matrix.
x = np.array([1.0, 2.0, 0.0, -1.0, 3.0, 1.0])
w = np.array([0.5, -1.0, 2.0])
n_out = len(x) - len(w) + 1
M = np.zeros((n_out, len(x)))
for i in range(n_out):
    M[i, i : i + len(w)] = w          # the same three numbers on every row
print("conv1d     :", conv1d(x, w))
print("matrix form:", M @ x)
print("matrix has", M.size, "entries but only", len(w), "free parameters")

# 2. Translation equivariance: shift the input, the output shifts too.
signal = np.zeros(12); signal[3] = 1.0; signal[4] = 2.0
shifted = np.roll(signal, 4)
print("\nresponse         :", conv1d(signal, w))
print("response, shifted:", conv1d(shifted, w))

# 3. A hand-made vertical-edge detector on a tiny image.
img = np.zeros((6, 6)); img[:, 3:] = 1.0      # dark left half, bright right half
sobel_x = np.array([[-1, 0, 1], [-2, 0, 2], [-1, 0, 1]], dtype=float)
print("\nimage:\n", img)
print("vertical edges:\n", conv2d(img, sobel_x))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The second experiment shows the property that makes convolutions right for images: translation equivariance. Shift the input and the output shifts by the same amount, unchanged otherwise. The detector responds to the pattern, not its address. The third experiment uses a classic hand-designed edge filter (the Sobel kernel); a trained network's first layer learns filters much like it on its own.

Seeing more of the image: depth, stride and pooling

One 3 × 3 layer sees a 3 × 3 window. Stack a second 3 × 3 layer on top and each of its outputs depends on a 5 × 5 window of the input; a third gives 7 × 7. This window is the receptive field, and with stride 1 it grows by k−1k - 1 per layer. To grow it faster, networks downsample: a stride-2 convolution or a pooling layer (taking the maximum or average of each 2 × 2 block) halves the height and width, so later layers see large regions cheaply. The typical convolutional network alternates convolutions and downsampling, trading spatial resolution for more channels as it goes deeper: early layers detect edges and colours, middle layers textures and parts, late layers whole objects.

Example— How much does weight sharing save?

Take a hidden layer of 56 × 56 positions with 64 channels, feeding a layer of 56 × 56 positions with 128 channels.

  • Convolution, 3 × 3 kernel: 3⋅3⋅64⋅128+128=73,8563 \cdot 3 \cdot 64 \cdot 128 + 128 = 73{,}856 parameters.
  • Fully connected: (56⋅56⋅64)×(56⋅56⋅128)=200,704×401,408≈8×1010(56 \cdot 56 \cdot 64) \times (56 \cdot 56 \cdot 128) = 200{,}704 \times 401{,}408 \approx 8 \times 10^{10} weights.

A factor of about a million. The compute is still substantial — each of the 56⋅56⋅128≈400,00056 \cdot 56 \cdot 128 \approx 400{,}000 outputs needs 3⋅3⋅64=5763 \cdot 3 \cdot 64 = 576 multiply-adds, about 231 million in total — but it is the kind of regular, parallel arithmetic that GPUs are built for (see Accelerators and distributed training).

Where convolutional networks went

The modern story starts with LeCun's networks for reading handwritten digits in the 1990s and became a revolution in 2012, when a deep convolutional network trained on GPUs (AlexNet) won the ImageNet image-recognition competition by a wide margin. Networks then grew deeper, until a 2015 change made depth easy: residual connections (He et al.), where each block computes x+F(x)x + F(x) instead of F(x)F(x). The identity path lets gradients flow straight through dozens or hundreds of layers. Remember it: every transformer block is built the same way.

Since 2020, vision transformers (Dosovitskiy et al.), which cut an image into patches and treat them like words, have taken over the largest vision and multimodal models. Convolutional networks have not disappeared: a careful modernisation (ConvNeXt) matched transformers at similar compute, convolutions are efficient on phones and embedded devices, and they remain common in audio, in scientific imaging, and as the "stem" that first turns pixels into patches inside many vision transformers.

IntuitionH

Recurrent networks: a summary carried through time

Now take a sequence x1,x2,…,xTx_1, x_2, \ldots, x_T — words, audio frames, daily temperatures. A recurrent neural network (RNN) reads it one element at a time and keeps a hidden state hth_t, a fixed-size vector meant to summarise everything so far:

ht=tanh⁡(Whht−1+Wxxt+b),h0=0.h_t = \tanh\left(W_h h_{t-1} + W_x x_t + b\right), \qquad h_0 = 0 .

The same WhW_h, WxW_x and bb are used at every step — weight sharing again, this time across time rather than space — so one network handles sequences of any length. An output layer reads hth_t to predict, for instance, the next word.

To train it, you unroll the recurrence into a deep feed-forward network with one layer per time step (all sharing weights) and run ordinary backpropagation through it. This is backpropagation through time.

Why the gradients vanish or explode

The loss at step TT depends on an early state hth_t through every step in between. By the chain rule,

∂hT∂ht=∏s=t+1Tdiag⁡ ⁣(1−hs2)Wh.\frac{\partial h_T}{\partial h_t} = \prod_{s=t+1}^{T} \operatorname{diag}\!\left(1 - h_s^2\right) W_h .

That is a product of T−tT - t matrices. Products of many matrices behave like powers: if the factors shrink vectors (largest singular value below 1), the product shrinks exponentially; if they stretch them, it grows exponentially. The tanh⁡\tanh derivative 1−h21 - h^2 is at most 1, so it only ever pushes towards shrinking. Run it:

python
import numpy as np

rng = np.random.default_rng(0)
d, T = 32, 60
Q, _ = np.linalg.qr(rng.normal(size=(d, d)))   # a random orthogonal matrix
xs = rng.normal(scale=0.5, size=(T, d))

def gradient_norms(scale):
    """Norm of d h_t / d h_0 for a tanh RNN whose recurrent matrix is scale * Q."""
    W_h = scale * Q
    h = np.zeros(d)
    J = np.eye(d)                                 # Jacobian d h_t / d h_0, starts as identity
    norms = []
    for t in range(T):
        h = np.tanh(W_h @ h + xs[t])
        J = (1 - h**2)[:, None] * (W_h @ J)       # chain rule: diag(tanh') @ W_h @ J
        norms.append(np.linalg.norm(J, 2))        # largest singular value
    return norms

for scale in [0.5, 1.0, 3.0]:
    n = gradient_norms(scale)
    print(f"scale {scale}: " + "  ".join(f"t={t}: {n[t-1]:.1e}" for t in [1, 10, 30, 60]))

# Without the tanh the story is pure linear algebra: the Jacobian is (scale * Q)^t,
# whose norm is exactly scale**t.
for scale in [0.9, 1.1]:
    print(f"linear RNN, scale {scale}: after 60 steps the gradient is multiplied by {scale**60:.1e}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With scale 0.5 the signal from step 0 has fallen to 10−2310^{-23} after 60 steps — numerically nothing. Even at scale 1.0 the tanh⁡\tanh saturation drains it to 10−710^{-7}. At scale 3 it explodes. There is no comfortable middle: the network cannot learn that a word at position 3 matters for a prediction at position 60, because almost no gradient signal connects them. This was analysed in the early 1990s by Hochreiter and by Bengio and colleagues, and revisited by Pascanu, Mikolov and Bengio.

Exploding gradients have a cheap fix: gradient clipping, rescaling the gradient whenever its norm exceeds a threshold. (Large models are still trained with clipping today.) Vanishing gradients need a change of architecture.

Gates: the LSTM idea

The Long Short-Term Memory network (Hochreiter and Schmidhuber, 1997) adds a second state, the cell ctc_t, updated additively:

ct=ft⊙ct−1+it⊙gt,ht=ot⊙tanh⁡(ct),c_t = f_t \odot c_{t-1} + i_t \odot g_t, \qquad h_t = o_t \odot \tanh(c_t),

where gtg_t is a candidate update and ft,it,ot∈(0,1)df_t, i_t, o_t \in (0, 1)^d are gates: sigmoid layers of xtx_t and ht−1h_{t-1} that decide, per coordinate, how much of the old cell to keep (forget gate), how much of the new candidate to write (input gate) and how much to expose (output gate). When the forget gate is near 1, ∂ct/∂ct−1≈1\partial c_t / \partial c_{t-1} \approx 1: information and gradient pass along the cell unchanged, like the identity path of a residual connection. The GRU (Cho et al., 2014) is a lighter variant with two gates and no separate cell.

Analogy, not a formal construction

A plain RNN is like passing a message along a line of people who each retell it in their own words: after twenty retellings little of the original survives. An LSTM gives the line a notebook that is passed along too. Each person may cross out a few lines or add a few, but whatever nobody touches arrives intact.

LSTMs powered speech recognition, machine translation and handwriting recognition through most of the 2010s. They still had two limits. In practice they remembered tens to a few hundred steps well, not thousands. And — this one matters most for modern hardware — they are sequential: you cannot compute h100h_{100} until you have h99h_{99}, so a long sequence cannot be processed in parallel during training. Attention removes both limits.

DefinitionsH

Embeddings: symbols as vectors

Neural networks do arithmetic. Words are not numbers. The simplest encoding gives word number ii in a vocabulary of size VV the one-hot vector ei∈RVe_i \in \mathbb{R}^V, all zeros except a 1 in place ii. But every two one-hot vectors are orthogonal: "tea" is exactly as far from "coffee" as from "volcano". Nothing learned about one word transfers to any other.

Definition— Embedding

An embedding is a learned map from a discrete set of VV symbols to vectors in Rd\mathbb{R}^d, with d≪Vd \ll V, stored as a matrix E∈RV×dE \in \mathbb{R}^{V \times d}. Symbol ii is represented by row ii of EE. Looking up a row is the same as multiplying the one-hot vector by the matrix, ei⊤Ee_i^{\top} E, so an embedding layer is just a linear layer applied to one-hot inputs, implemented as a table lookup.

Similarity between embeddings is usually measured by the angle between them, the cosine similarity

cos⁡(u,v)=u⋅v∥u∥ ∥v∥∈[−1,1],\cos(u, v) = \frac{u \cdot v}{\lVert u \rVert \, \lVert v \rVert} \in [-1, 1],

which ignores length (often related to word frequency) and keeps direction.

Where the vectors come from: the distributional hypothesis

The idea, associated with the linguists Zellig Harris and J. R. Firth in the 1950s, is that words that occur in similar contexts tend to have similar meanings. "Tea" and "coffee" both follow "drinks hot" and precede "every morning"; "bus" and "train" both "leave the station". If we describe each word by the company it keeps, similar words get similar descriptions.

The oldest way to do this is to count. Build a matrix of how often each word appears next to each other word, reweight the counts to emphasise surprising pairs (positive pointwise mutual information, PPMI), and compress each long, sparse row into a short dense vector with the singular value decomposition (see Advanced linear algebra):

python
import numpy as np

corpus = """
amara drinks hot tea every morning . kenji drinks hot coffee every morning .
lucia pours sweet tea for guests . omar pours sweet coffee for guests .
the bus leaves the station at noon . the train leaves the station at dawn .
we took the train across the country . we took the bus across the city .
the cat sleeps on the warm mat . the dog sleeps on the warm rug .
a dog chased the ball in the park . a cat chased the mouse in the garden .
priya drinks cold juice after school . the bus stops near the school .
""".split()

vocab = sorted(set(corpus))
idx = {w: i for i, w in enumerate(vocab)}
V = len(vocab)

# 1. Count how often each pair of words appears side by side.
window = 1
C = np.zeros((V, V))
for i, w in enumerate(corpus):
    for j in range(max(0, i - window), min(len(corpus), i + window + 1)):
        if j != i:
            C[idx[w], idx[corpus[j]]] += 1

# 2. Positive pointwise mutual information: how much more often than chance?
total = C.sum()
p_w = C.sum(axis=1, keepdims=True) / total
p_c = C.sum(axis=0, keepdims=True) / total
with np.errstate(divide="ignore"):
    pmi = np.log((C / total) / (p_w * p_c))
ppmi = np.maximum(pmi, 0)

# 3. Compress each 1-of-V row to a dense 8-dimensional vector with the SVD.
U, S, _ = np.linalg.svd(ppmi)
E = U[:, :8] * S[:8]                       # the embedding matrix: one row per word

def cosine(a, b):
    return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

def neighbours(word, k=3):
    v = E[idx[word]]
    sims = [(cosine(v, E[i]), w) for i, w in enumerate(vocab) if w != word]
    return sorted(sims, reverse=True)[:k]

for word in ["tea", "train", "cat"]:
    print(word, "->", ", ".join(f"{w} ({s:.2f})" for s, w in neighbours(word)))
print(f"\ncos(tea, coffee) = {cosine(E[idx['tea']], E[idx['coffee']]):.2f}")
print(f"cos(tea, train)  = {cosine(E[idx['tea']], E[idx['train']]):.2f}")
print("with one-hot vectors both would be exactly 0: every pair of words equally unrelated")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Nobody told the program that tea and coffee are both drinks. It found that out from where they appear. With fifteen sentences the other neighbours are noisy; with billions of words the same method gives good vectors.

word2vec: predict, don't count

word2vec (Mikolov and colleagues at Google, 2013) learns the vectors by prediction instead. In its skip-gram form each word ww has an input vector vwv_w and an output vector uwu_w, and training makes σ(uc⋅vw)\sigma(u_c \cdot v_w) large for words cc actually seen near ww and small for a handful of randomly drawn "negative" words (negative sampling). It is a logistic regression on word pairs, trained by stochastic gradient descent over a huge corpus — cheap enough to run on billions of words in 2013. Later work showed that skip-gram with negative sampling is closely related to factorising a shifted PMI matrix, so counting and predicting turned out to be two routes to similar vectors. GloVe (Stanford, 2014) made that connection explicit.

The famous party trick is vector arithmetic: the vector for "king" minus "man" plus "woman" lands near "queen". Treat it with care: such analogies work for some relations and not others, and the standard evaluation quietly excludes the input words from the answers, which flatters the results. More importantly, embeddings learn all the regularities of their text, including stereotypes: Bolukbasi et al. (2016) found occupational gender bias in widely used word vectors. A model is a mirror of its data before it is anything else.

Embeddings today

Every language model in this course starts with an embedding table: the first layer maps each token id to a vector of a few thousand dimensions. Two things changed after word2vec. First, the units are now subword tokens rather than words (Language models). Second, the vector from the table is only the starting point: each layer of a transformer updates it using the surrounding context, so "bank" in "river bank" and in "bank loan" ends up with different contextual representations. Separately trained embedding models, which map a whole sentence or document to one vector, are the backbone of semantic search and retrieval-augmented generation (Reasoning models, tools and agents).

ExplanationH

Sequence to sequence, and the bottleneck

Machine translation maps a sequence to another sequence of a different length. In 2014 Sutskever, Vinyals and Le and Cho et al. proposed the encoder–decoder (or seq2seq) design:

  1. An encoder RNN reads the source sentence, word by word, and its final hidden state hTh_T becomes the context vector cc.
  2. A decoder RNN starts from cc and generates the target sentence one word at a time, each step feeding back the word it just produced, until it emits an end-of-sentence token.

Trained end to end on pairs of sentences, it worked remarkably well, and within two years neural translation systems had replaced the older phrase-based ones in production.

But look at step 1. Everything the decoder will ever know about the source sentence must pass through one fixed-size vector cc — say 1,000 numbers — whether the sentence has five words or fifty. That is the bottleneck. Researchers measured the consequence directly: translation quality dropped as sentences got longer. Reversing the source sentence, a trick Sutskever and colleagues found helpful, eased it by putting the first source words close to the first target words; it did not remove it.

Bahdanau, Cho and Bengio (2014) proposed the fix. Keep all the encoder's hidden states h1,…,hTh_1, \ldots, h_T, not just the last. When the decoder produces target word tt, let it compute a relevance score between its own state and each hjh_j, turn the scores into weights with a softmax, and read a weighted average of the hjh_j — a different context vector for every output word. Translating "the black cat" into Swahili or Spanish, the decoder can look mostly at "cat" when it writes the noun and at "black" when it writes the adjective, wherever they sit in the source. The model learned these soft alignments by itself.

They called it attention. Three years later a team at Google asked what happens if you keep the attention and throw away the recurrence entirely. That is the next lesson.

Common misconceptionsH

Common misconceptions

Misconception

"A convolutional layer flips its kernel, as in signal processing." Deep-learning libraries compute cross-correlation, without the flip. Because the kernel is learned, the two are equivalent; the distinction matters only when you load hand-designed kernels or compare with a signal-processing textbook.

Misconception

"CNNs are translation invariant." Convolutions are translation equivariant: shifting the input shifts the feature map. Invariance (the output not changing at all) comes only approximately, from pooling and the final global average, and real networks are measurably sensitive to small shifts because strided downsampling breaks exact equivariance.

Misconception

"LSTMs solved long-range dependencies." Gating greatly eased the vanishing gradient, but LSTMs still struggled beyond a few hundred steps and could not be parallelised over time. Both facts motivated attention.

Misconception

"Recurrent networks are obsolete." Pure RNNs are rare in large language models, but recurrence has returned in new forms. State-space models such as Mamba and related linear-attention layers keep a fixed-size state, run in time linear in sequence length, and can be trained in parallel. Several labs have released hybrid models that interleave a few attention layers with many of these recurrent layers; whether hybrids will displace pure transformers at the frontier is an open, actively tested question as of 2026.

Misconception

"Word embeddings capture meaning." They capture distributional statistics — which words keep company with which — and that correlates with meaning surprisingly well. It also captures the biases, gaps and quirks of the corpus, and says nothing about words that are rare in it.

Exercises

Exercises

1. Convolutions, sequences and embeddings

Warm-upHuman

A 1-D convolution has input length n=32n = 32, kernel size k=5k = 5, stride s=2s = 2 and padding p=2p = 2 on each side. What is the output length?

2. Convolutions, sequences and embeddings

Warm-upHuman

A 2-D convolutional layer has a 3×33 \times 3 kernel, 32 input channels and 64 output channels, with one bias per output channel. How many parameters does it have?

3. Convolutions, sequences and embeddings

CoreHuman

Four 3×33 \times 3 convolutional layers with stride 1 and no pooling are stacked. One output position of the fourth layer depends on a square window of the original input. What is the side length of that window?

4. Convolutions, sequences and embeddings

CoreComputational

Write conv2d_multi(x, w, b) for a full convolutional layer (stride 1, no padding).

  • x has shape (C_in, H, W).
  • w has shape (C_out, C_in, kh, kw).
  • b has shape (C_out,).
  • Return an array of shape (C_out, H - kh + 1, W - kw + 1) where output channel o at position (i, j) is b[o] plus the sum over all input channels and kernel positions of w[o, c, u, v] * x[c, i + u, j + v].

Use numpy; loops over output positions are fine.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Convolutions, sequences and embeddings

CoreComputational

Write rnn_forward(xs, W_x, W_h, b) for a vanilla RNN.

  • xs has shape (T, n_in): one input vector per time step.
  • W_x has shape (d, n_in), W_h has shape (d, d), b has shape (d,).
  • Start from h0=0h_0 = 0 and apply ht=tanh⁡(Whht−1+Wxxt+b)h_t = \tanh(W_h h_{t-1} + W_x x_t + b) for t=1,…,Tt = 1, \ldots, T.
  • Return an array of shape (T, d) whose row t−1t-1 is hth_t.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Convolutions, sequences and embeddings

CoreComputational

Write most_similar(E, vocab, word, k).

  • E is an embedding matrix of shape (V, d); row i is the vector of vocab[i].
  • Return a list of the k words (strings) with the highest cosine similarity to word, most similar first, excluding word itself.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Convolutions, sequences and embeddings

CoreHuman

Why does a vanilla RNN struggle to learn that an input 100 steps ago affects the current output?

Choose one

8. Convolutions, sequences and embeddings

CoreHuman

In a 2014-style encoder–decoder translation model, what exactly is the bottleneck, and what did attention change?

Choose one

Further reading and resources