Skip to content
LumenaryAcademy
ML 90 min

Neural networks and backpropagation

Stacking simple units into flexible functions, and computing every gradient in one backward sweep — built from scratch.

Code tested · 7AI-draftedAwaiting review by a computer scientist

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Beyond straight lines

The models of Learning from data are linear: logistic regression separates classes with a straight line (or flat hyperplane). Many real boundaries are curved. The simplest example is XOR: label a point 1 if exactly one of its two binary coordinates is 1. The points (0,1)(0,1) and (1,0)(1,0) are positive, (0,0)(0,0) and (1,1)(1,1) negative, and no single line separates them.

One fix is to engineer features by hand — add x1x2x_1 x_2, or x12x_1^2 — and keep the model linear in them. That works when you know which features matter. For images, sound or text, nobody does. Neural networks learn the features too: they compose many simple, adjustable units so that early layers compute useful features and later layers combine them. This lesson builds them from the ground up, together with backpropagation, the algorithm that computes their gradients efficiently and makes training them possible at all.

DefinitionsH

Neurons and layers

Definition— Neuron (unit)

A function of an input vector x∈Rd\mathbf{x} \in \mathbb{R}^d with weights w∈Rd\mathbf{w} \in \mathbb{R}^d, a bias bb and a fixed nonlinear activation function ϕ\phi:   a=ϕ(w⊤x+b)\;a = \phi(\mathbf{w}^\top\mathbf{x} + b). The quantity z=w⊤x+bz = \mathbf{w}^\top\mathbf{x} + b is its pre-activation.

Logistic regression is a single neuron with ϕ=σ\phi = \sigma. A layer of mm neurons applied to the same input computes a=ϕ(Wx+b)\mathbf{a} = \phi(W\mathbf{x} + \mathbf{b}), with a weight matrix W∈Rm×dW \in \mathbb{R}^{m \times d} and ϕ\phi applied elementwise. A multilayer perceptron (MLP), or fully connected feedforward network, composes layers:

h1=ϕ(W1x+b1),h2=ϕ(W2h1+b2),…,y^=WLhL−1+bL.\mathbf{h}_1 = \phi(W_1 \mathbf{x} + \mathbf{b}_1), \quad \mathbf{h}_2 = \phi(W_2 \mathbf{h}_1 + \mathbf{b}_2), \quad \dots, \quad \hat{y} = W_L \mathbf{h}_{L-1} + \mathbf{b}_L .

The intermediate hℓ\mathbf{h}_\ell are hidden layers; the number of units in each is its width, the number of layers the network's depth. The last layer is usually linear, followed by a sigmoid or softmax inside the loss for classification. A layer from dd inputs to mm outputs has dm+mdm + m parameters, so a 784-128-10 network for digit images has 784⋅128+128+128⋅10+10=101,770784 \cdot 128 + 128 + 128 \cdot 10 + 10 = 101{,}770 of them.

The nonlinearity is essential:

Theorem— Without activations, depth adds nothing

A composition of affine maps is an affine map. So a network whose layers all use the identity activation computes a function of the form Wx+bW\mathbf{x} + \mathbf{b}, no matter how many layers it has.

Proof

For two layers, W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2)W_2(W_1\mathbf{x} + \mathbf{b}_1) + \mathbf{b}_2 = (W_2W_1)\mathbf{x} + (W_2\mathbf{b}_1 + \mathbf{b}_2), which is affine with matrix W2W1W_2W_1 and bias W2b1+b2W_2\mathbf{b}_1 + \mathbf{b}_2. Induction on the number of layers finishes the proof. ∎

∎

IntuitionH

Activation functions

ActivationFormulaDerivativeNotes
Sigmoidσ(z)=1/(1+e−z)\sigma(z) = 1/(1 + e^{-z})σ(z)(1−σ(z))≤14\sigma(z)(1 - \sigma(z)) \le \tfrac14outputs in (0,1)(0, 1); saturates for large ∣z∣\lvert z\rvert
Tanhtanh⁡z\tanh z1−tanh⁡2z1 - \tanh^2 zzero-centred, outputs in (−1,1)(-1, 1); also saturates
ReLUmax⁡(0,z)\max(0, z)11 if z>0z > 0, else 00cheap; no saturation for z>0z > 0; units can "die"
GELUz Φ(z)z\,\Phi(z)Φ(z)+z φ(z)\Phi(z) + z\,\varphi(z)a smooth ReLU; Φ\Phi, φ\varphi are the standard normal CDF and density

Sigmoid and tanh were the classic choices, but both saturate: for large ∣z∣\lvert z \rvert their derivative is nearly zero, and since backpropagation multiplies derivatives layer by layer, gradients in deep sigmoid networks shrink towards nothing — the vanishing gradient problem. The sigmoid's derivative never exceeds 1/41/4, so ten sigmoid layers can scale a gradient by 4−104^{-10}, about one millionth, even at best.

The rectified linear unit (ReLU) changed deep learning in the early 2010s: its derivative is exactly 1 wherever it is active, so gradients pass through unchanged. Its flaw is that a unit whose pre-activation is negative for every input has zero gradient and never recovers (a dead unit). GELU weights its input by the probability that a standard normal variable is below it; it behaves like ReLU for large ∣z∣\lvert z \rvert but is smooth near zero. GELU was used in BERT and GPT-2, and smooth, gated relatives (such as the SwiGLU unit) are common in today's large language models. In practice the choice among the modern options matters less than initialisation, normalisation and the learning rate — but the choice of some nonlinearity is essential.

DefinitionsH

What one hidden layer can do: universal approximation

How expressive are these networks? The classic answer is a theorem proved for sigmoids by George Cybenko (1989) and Kurt Hornik (1991), and in a general form by Leshno, Lin, Pinkus and Schocken (1993).

Theorem— Universal approximation (one hidden layer)

Let ϕ:R→R\phi : \mathbb{R} \to \mathbb{R} be continuous and not a polynomial. Let K⊂RdK \subset \mathbb{R}^d be compact (closed and bounded), f:K→Rf : K \to \mathbb{R} continuous, and ε>0\varepsilon > 0. Then there exist a number of units NN and parameters ck,bk∈Rc_k, b_k \in \mathbb{R}, wk∈Rd\mathbf{w}_k \in \mathbb{R}^d such that

g(x)=∑k=1Nck ϕ(wk⊤x+bk)satisfiessup⁡x∈K∣f(x)−g(x)∣<ε.g(\mathbf{x}) = \sum_{k=1}^{N} c_k\, \phi(\mathbf{w}_k^\top \mathbf{x} + b_k) \quad\text{satisfies}\quad \sup_{\mathbf{x} \in K} \lvert f(\mathbf{x}) - g(\mathbf{x}) \rvert < \varepsilon .

The proof needs functional analysis (the Hahn–Banach and Riesz representation theorems in Cybenko's version), but in one dimension the idea is visible with ReLUs: each unit ckmax⁡(0,x−tk)c_k \max(0, x - t_k) adds a hinge that changes the slope by ckc_k at the point tkt_k, so a sum of hinges can follow any continuous curve as closely as you like, piece by linear piece. The block below sets the weights by hand to interpolate a wiggly function and measures the worst error as units are added.

python
import numpy as np

# One hidden layer of ReLU units, with weights set by hand rather than learned:
# each unit adds a "hinge" that changes the slope at one point, so the network
# is the piecewise-linear interpolation of the target at evenly spaced knots.
def relu(z):
    return np.maximum(z, 0)

def hinge_network(f, a, b, units):
    knots = np.linspace(a, b, units + 1)
    values = f(knots)
    slopes = np.diff(values) / np.diff(knots)
    # output(x) = values[0] + sum_k c_k * relu(x - knots[k]), with c_k the change in slope at knot k
    c = np.concatenate([[slopes[0]], np.diff(slopes)])
    def net(x):
        hidden = relu(x[:, None] - knots[None, :-1])     # one column per hidden unit
        return values[0] + hidden @ c                     # a linear output layer
    return net

target = lambda x: np.sin(3 * x) + 0.5 * x
x = np.linspace(-2, 2, 2001)
print("hidden units   worst error on [-2, 2]")
for units in (2, 4, 8, 16, 32, 64, 128):
    net = hinge_network(target, -2, 2, units)
    print(f"{units:12d}   {np.max(np.abs(net(x) - target(x))):.5f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Each doubling of the width cuts the error by about four, as you would expect from piecewise-linear interpolation of a smooth function. Now be careful about what the theorem does not say:

  • It says nothing about size. NN can be astronomically large; for some functions in dd dimensions, shallow networks need a number of units exponential in dd. Depth helps: Telgarsky (2016) constructed functions that deep ReLU networks represent with few units but shallow ones need exponentially many to approximate.
  • It says nothing about learning. The weights exist; the theorem does not say gradient descent finds them.
  • It says nothing about generalisation. Approximating a function well on all of KK is different from learning it from finitely many noisy examples, which is the problem of the previous lesson.

So "neural networks can represent anything" is true only in this limited sense. Why they learn so well in practice is a deeper and still partly open question.

IntuitionH

Computational graphs

To train a network by gradient descent we need ∂L/∂θ\partial L / \partial \theta for every parameter — possibly billions of them. The key is to see the computation as a graph.

A computational graph is a directed acyclic graph whose nodes are values and whose edges say which values were computed from which. Inputs and parameters are leaves; each other node is an elementary operation (add, multiply, tanh, …) applied to its parents; the loss is the final node. Here is the graph of a one-example squared error for a single linear unit, L=(wx+b−y)2L = (wx + b - y)^2, with the numbers w=0.5w = 0.5, b=−1b = -1, x=3x = 3, y=2y = 2.

×+−( )²w = 0.5x = 3b = −1y = 2u = 1.5v = 0.5r = −1.5L = 2.25−3−3−31∂L/∂w = −9∂L/∂b = −3
The computational graph of L = (wx + b − y)². Forward values are written above the edges; the gradients ∂L/∂(value) computed by the backward pass are written below, in colour.

The forward pass evaluates the nodes from left to right. The backward pass goes right to left, computing for every node vv its adjoint vˉ=∂L/∂v\bar v = \partial L / \partial v.

DefinitionsH

The chain rule and backpropagation

In one variable, the chain rule says that if LL depends on uu and uu on ww, then dLdw=dLdududw\frac{dL}{dw} = \frac{dL}{du}\frac{du}{dw}. In a graph, a value can influence the loss along several paths, and the multivariable chain rule adds them up: for a node vv whose value is used by children c1,…,ckc_1, \dots, c_k,

vˉ=∂L∂v=∑j=1kcˉj ∂cj∂v.\bar v = \frac{\partial L}{\partial v} = \sum_{j=1}^{k} \bar c_j \, \frac{\partial c_j}{\partial v}.

Each ∂cj/∂v\partial c_j / \partial v is a local derivative, known from the single operation that computed cjc_j: for c=v⋅wc = v \cdot w it is ww; for c=v+wc = v + w it is 1; for c=tanh⁡vc = \tanh v it is 1−c21 - c^2. Backpropagation is just this formula applied to every node in reverse topological order, so that every child's adjoint is ready before it is needed. Start with Lˉ=1\bar L = 1.

Example— Backpropagation by hand

In the figure: Lˉ=1\bar L = 1. Since L=r2L = r^2, rˉ=2r⋅Lˉ=−3\bar r = 2r \cdot \bar L = -3. Since r=v−yr = v - y, vˉ=rˉ⋅1=−3\bar v = \bar r \cdot 1 = -3. Since v=u+bv = u + b, uˉ=−3\bar u = -3 and bˉ=−3\bar b = -3. Since u=wxu = wx, wˉ=uˉ⋅x=−9\bar w = \bar u \cdot x = -9 (and xˉ=uˉ⋅w=−1.5\bar x = \bar u \cdot w = -1.5, which we would not need unless xx were a parameter). Gradient descent would therefore increase ww and bb, pushing the prediction wx+b=0.5wx + b = 0.5 up towards y=2y = 2.

Why backwards? Forward-mode differentiation pushes derivatives from one input towards the outputs, so it costs one pass per input — a billion passes for a billion parameters. Reverse mode pulls derivatives from one output back to all inputs at once, so one backward pass gives the whole gradient of a scalar loss, at a cost of a small constant multiple of the forward pass. That asymmetry is why backpropagation made large networks trainable. Its price is memory: the backward pass needs the intermediate values of the forward pass, so training must store activations — the main reason training needs far more memory than running a model.

Backpropagation, then, is not a learning rule: it is an efficient way to compute gradients. Gradient descent (or the optimisers of the next lesson) does the learning.

ComputationC

A tiny automatic-differentiation engine

Frameworks such as PyTorch and JAX implement reverse mode for arrays. The essential idea fits in a page for scalars. In the design below, every operation computes its result and its local derivatives during the forward pass and stores them with the result; backward then only has to sort the graph and apply the sum-over-children formula. The block checks the engine against the hand calculation and against finite differences, then uses it to train a tiny network on XOR.

python
import math
import random

class Var:
    """A scalar that remembers how it was computed. Each Var stores, for every input it was
    computed from, the local derivative d(self)/d(input), worked out during the forward pass."""

    def __init__(self, value, inputs=()):
        self.value = value
        self.inputs = inputs          # tuple of (input Var, local derivative)
        self.grad = 0.0

    # Each operation computes its value AND its local derivatives, then wraps them.
    def __add__(self, other):
        other = other if isinstance(other, Var) else Var(other)
        return Var(self.value + other.value, ((self, 1.0), (other, 1.0)))

    def __mul__(self, other):
        other = other if isinstance(other, Var) else Var(other)
        return Var(self.value * other.value, ((self, other.value), (other, self.value)))

    def __neg__(self):
        return self * -1.0

    def __sub__(self, other):
        return self + (-other if isinstance(other, Var) else Var(-other))

    def __pow__(self, k):             # k is a plain number
        return Var(self.value ** k, ((self, k * self.value ** (k - 1)),))

    __radd__ = __add__
    __rmul__ = __mul__

    def tanh(self):
        t = math.tanh(self.value)
        return Var(t, ((self, 1 - t * t),))

    def relu(self):
        return Var(max(self.value, 0.0), ((self, 1.0 if self.value > 0 else 0.0),))

    def exp(self):
        e = math.exp(self.value)
        return Var(e, ((self, e),))

    def log(self):
        return Var(math.log(self.value), ((self, 1 / self.value),))

    def backward(self):
        """Reverse-mode differentiation: fill .grad = d(self)/d(v) for every v this depends on."""
        order, seen = [], set()
        def visit(v):                 # topological order: inputs before the values built from them
            if id(v) not in seen:
                seen.add(id(v))
                for parent, _ in v.inputs:
                    visit(parent)
                order.append(v)
        visit(self)
        for v in order:
            v.grad = 0.0
        self.grad = 1.0
        for v in reversed(order):     # outputs first: each node passes its gradient back
            for parent, local in v.inputs:
                parent.grad += local * v.grad     # chain rule, summed over every path

# 1. The worked example from the text: L = (w*x + b - y)^2
w, b = Var(0.5), Var(-1.0)
x, y = 3.0, 2.0
L = (w * x + b - y) ** 2
L.backward()
print(f"L = {L.value}, dL/dw = {w.grad}, dL/db = {b.grad}")

# 2. Check against finite differences on a messier expression.
def f(a, c):
    return ((a * c).tanh() + (a ** 2) * c.exp()).log()
a, c = Var(0.7), Var(-0.3)
out = f(a, c)
out.backward()
h = 1e-6
num_a = (f(Var(0.7 + h), Var(-0.3)).value - f(Var(0.7 - h), Var(-0.3)).value) / (2 * h)
num_c = (f(Var(0.7), Var(-0.3 + h)).value - f(Var(0.7), Var(-0.3 - h)).value) / (2 * h)
print(f"autograd: {a.grad:.8f} {c.grad:.8f}")
print(f"numeric:  {num_a:.8f} {num_c:.8f}")

# 3. Train a 2-3-1 network on XOR, which no linear model can fit.
random.seed(1)
W1 = [[Var(random.uniform(-1, 1)) for _ in range(2)] for _ in range(3)]
b1 = [Var(0.0) for _ in range(3)]
W2 = [Var(random.uniform(-1, 1)) for _ in range(3)]
b2 = Var(0.0)
params = [p for row in W1 for p in row] + b1 + W2 + [b2]

def predict(x1, x2):
    hidden = [(W1[j][0] * x1 + W1[j][1] * x2 + b1[j]).tanh() for j in range(3)]
    return sum((W2[j] * hidden[j] for j in range(3)), b2)

data = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]
for step in range(301):
    loss = sum(((predict(*xs) - t) ** 2 for xs, t in data), Var(0.0)) * 0.25
    loss.backward()
    for p in params:
        p.value -= 0.2 * p.grad
    if step % 100 == 0:
        print(f"step {step:3d}: loss {loss.value:.4f}")
print("XOR predictions:", [round(predict(*xs).value, 2) for xs, _ in data])
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

About fifty lines of Python, and the network learns XOR — something no linear model can do. The structure is exactly that of an industrial framework: a graph recorded during the forward pass, local derivatives per operation, a reverse sweep accumulating adjoints. What frameworks add is operations on whole tensors (so one node is a matrix multiply, not one multiplication), GPU kernels, and many engineering refinements.

ComputationC

Backpropagation for a whole layer, in numpy

A scalar engine creates one node per multiplication, far too slow for real networks. For an MLP we can apply the chain rule to whole matrices once, by hand. Take a batch of nn examples as the rows of XX, one hidden ReLU layer and a sigmoid output with cross-entropy:

Z1=XW1+b1,A1=ReLU⁡(Z1),z=A1W2+b2,L=1n∑iℓ(σ(zi),yi).Z_1 = XW_1 + \mathbf{b}_1, \quad A_1 = \operatorname{ReLU}(Z_1), \quad \mathbf{z} = A_1 W_2 + b_2, \quad L = \frac{1}{n}\sum_i \ell(\sigma(z_i), y_i).

Going backwards, with Qˉ\bar Q meaning ∂L/∂Q\partial L / \partial Q and ⊙\odot elementwise multiplication:

zˉ=1n(σ(z)−y),Wˉ2=A1⊤zˉ,bˉ2=∑izˉi,Aˉ1=zˉ W2⊤,\bar{\mathbf{z}} = \tfrac{1}{n}(\sigma(\mathbf{z}) - \mathbf{y}), \quad \bar W_2 = A_1^\top \bar{\mathbf{z}}, \quad \bar b_2 = \textstyle\sum_i \bar z_i, \quad \bar A_1 = \bar{\mathbf{z}}\, W_2^\top,

Zˉ1=Aˉ1⊙1[Z1>0],Wˉ1=X⊤Zˉ1,bˉ1=∑i(Zˉ1)i,:.\bar Z_1 = \bar A_1 \odot \mathbb{1}[Z_1 > 0], \quad \bar W_1 = X^\top \bar Z_1, \quad \bar{\mathbf{b}}_1 = \textstyle\sum_i (\bar Z_1)_{i,:}.

Each line is the scalar rule in matrix form, and a dimension check catches most mistakes: Wˉ1\bar W_1 must have the shape of W1W_1, and X⊤Zˉ1X^\top \bar Z_1 does. The first line is the same "prediction minus truth" you met in logistic regression. The code checks the formulas numerically before training a network on the "two moons" dataset, then draws the boundary it learned.

python
import numpy as np

rng = np.random.default_rng(0)

def moons(n, noise=0.15):
    """Two interleaving half-circles: a classic dataset no straight line can separate."""
    t = rng.uniform(0, np.pi, n)
    upper = np.column_stack([np.cos(t), np.sin(t)])
    lower = np.column_stack([1 - np.cos(t), 0.5 - np.sin(t)])
    X = np.vstack([upper, lower]) + rng.normal(0, noise, (2 * n, 2))
    y = np.concatenate([np.zeros(n), np.ones(n)])
    return X, y

X, y = moons(200)
X_test, y_test = moons(200)

# A 2 -> 32 -> 1 network: ReLU hidden layer, sigmoid output, binary cross-entropy loss.
H = 32
params = {
    "W1": rng.normal(0, np.sqrt(2 / 2), (2, H)),       # He initialisation (see below)
    "b1": np.zeros(H),
    "W2": rng.normal(0, np.sqrt(1 / H), (H, 1)),
    "b2": np.zeros(1),
}

def forward(p, X):
    Z1 = X @ p["W1"] + p["b1"]
    A1 = np.maximum(Z1, 0)
    Z2 = A1 @ p["W2"] + p["b2"]
    return Z1, A1, Z2[:, 0]

def loss_and_grads(p, X, y):
    n = len(y)
    Z1, A1, z = forward(p, X)
    loss = np.mean(np.logaddexp(0, -z) * y + np.logaddexp(0, z) * (1 - y))
    # Backward pass: the chain rule, layer by layer, from the loss back to the inputs.
    dz = (0.5 * (1 + np.tanh(z / 2)) - y) / n          # dL/dz for sigmoid + cross-entropy
    grads = {"W2": A1.T @ dz[:, None], "b2": np.array([dz.sum()])}
    dA1 = dz[:, None] @ p["W2"].T
    dZ1 = dA1 * (Z1 > 0)                               # ReLU passes gradient only where active
    grads["W1"] = X.T @ dZ1
    grads["b1"] = dZ1.sum(axis=0)
    return loss, grads

# Check the hand-derived gradients against finite differences before trusting them.
loss, grads = loss_and_grads(params, X, y)
for name, idx in [("W1", (1, 3)), ("b1", (5,)), ("W2", (7, 0)), ("b2", (0,))]:
    old = params[name][idx]
    params[name][idx] = old + 1e-5; up = loss_and_grads(params, X, y)[0]
    params[name][idx] = old - 1e-5; down = loss_and_grads(params, X, y)[0]
    params[name][idx] = old
    print(f"d loss / d {name}{list(idx)}: backprop {grads[name][idx]: .7f}  numeric {(up - down) / 2e-5: .7f}")

def accuracy(p, X, y):
    return np.mean((forward(p, X)[2] > 0) == y)

for step in range(3001):
    loss, grads = loss_and_grads(params, X, y)
    for name in params:
        params[name] -= 0.5 * grads[name]
    if step % 1000 == 0:
        print(f"step {step:4d}: loss {loss:.4f}, train accuracy {accuracy(params, X, y):.1%}")
print(f"test accuracy: {accuracy(params, X_test, y_test):.1%}")

# The learned decision regions, drawn in text: '#' where the network predicts class 1.
for row in np.linspace(1.4, -0.9, 11):
    cols = np.linspace(-1.4, 2.4, 48)
    grid = np.column_stack([cols, np.full_like(cols, row)])
    print("".join("#" if z > 0 else "." for z in forward(params, grid)[2]))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The network bends its boundary between the two moons, which no linear classifier could do. Note the habit on display: check hand-written gradients numerically before trusting them. A wrong gradient often still decreases the loss a little, so a bug can hide for a long time.

ComputationC

Initialisation

Where training starts matters more than you might expect.

Break the symmetry. If every weight starts at the same value (zero, say), every hidden unit in a layer computes the same function, receives the same gradient, and stays identical forever: a hundred units behave as one. Weights must start random. Biases can start at zero.

Get the scale right. Consider one unit z=∑i=1ninwihiz = \sum_{i=1}^{n_{\text{in}}} w_i h_i with independent zero-mean weights of variance s2s^2, independent of the inputs. Then Var⁡(z)=nin s2 E[hi2]\operatorname{Var}(z) = n_{\text{in}}\, s^2\, \mathbb{E}[h_i^2]. If nins2n_{\text{in}} s^2 is above the "right" value, the signal grows layer after layer; below it, the signal shrinks — exponentially in depth either way, and the gradients do the same on the way back. For tanh near zero, which is roughly linear, keeping the scale steady needs s2=1/nins^2 = 1/n_{\text{in}}; Glorot and Bengio (2010) proposed s2=2/(nin+nout)s^2 = 2/(n_{\text{in}} + n_{\text{out}}) to balance the forward and backward passes. ReLU zeroes half of its inputs on average, halving E[h2]\mathbb{E}[h^2], so He and colleagues (2015) doubled it: s2=2/nins^2 = 2/n_{\text{in}}.

python
import numpy as np

rng = np.random.default_rng(0)
width, depth = 256, 20
x = rng.normal(0, 1, (1000, width))                    # a batch of standardised inputs

def activation_spread(act, weight_std):
    """Push the batch through `depth` layers; report the std of activations at some layers."""
    h, report = x, {}
    for layer in range(1, depth + 1):
        W = rng.normal(0, weight_std, (width, width))
        h = act(h @ W)
        if layer in (1, 5, 10, 20):
            report[layer] = h.std()
    return report

relu = lambda z: np.maximum(z, 0)
settings = [
    ("ReLU, std 0.01", relu, 0.01),
    ("ReLU, std 1 / sqrt(fan_in)", relu, np.sqrt(1 / width)),
    ("ReLU, He: sqrt(2 / fan_in)", relu, np.sqrt(2 / width)),
    ("ReLU, std 0.1", relu, 0.1),
    ("tanh, std 0.01", np.tanh, 0.01),
    ("tanh, Xavier: sqrt(1 / fan_in)", np.tanh, np.sqrt(1 / width)),
]
print(f"{'initialisation':32s}" + "".join(f"{'layer ' + str(k):>12s}" for k in (1, 5, 10, 20)))
for name, act, std in settings:
    r = activation_spread(act, std)
    print(f"{name:32s}" + "".join(f"{r[k]:12.3g}" for k in (1, 5, 10, 20)))
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With He initialisation, the activations of a 20-layer ReLU network keep roughly the same size all the way down. Too small, and they vanish within a few layers; slightly too large, and they grow without bound. These rules assume particular activations and layer types, and modern networks also use normalisation layers and residual connections (next lessons) that make training much less sensitive to the exact scale. But a bad initialisation can still stop a network from learning at all, and frameworks choose these defaults for you for good reason.

History & global context

A short history

Frank Rosenblatt's perceptron (1958) was a single trainable neuron with a threshold activation. Marvin Minsky and Seymour Papert's book Perceptrons (1969) analysed the limits of single-layer networks, XOR among them, and interest waned. Reverse-mode differentiation was described by Seppo Linnainmaa in 1970 and proposed for neural networks by Paul Werbos in 1974; the 1986 paper by Rumelhart, Hinton and Williams showed convincingly that backpropagation lets multilayer networks learn useful internal representations, and it spread widely. Yann LeCun's convolutional networks read handwritten postcodes around 1990 and bank cheques later that decade. Networks then went out of fashion again until GPUs, large labelled datasets and techniques like ReLU made very deep networks trainable; the turning point is usually dated to 2012, when the AlexNet network won the ImageNet image-recognition competition by a wide margin. The rest of this level, and the next, follow from there.

Common misconceptionsH

Common misconceptions

Misconception

"The universal approximation theorem explains why deep learning works." It says a large enough network can represent any continuous function on a compact set. It says nothing about how large, whether training finds the weights, or whether the result generalises from data.

Misconception

"Backpropagation is how neural networks learn." Backpropagation computes gradients. An optimiser such as gradient descent uses them to change the parameters. They are separate pieces, and the same backpropagation serves many optimisers.

Misconception

"Artificial neurons work like biological neurons." The name is historical inspiration. A unit is a weighted sum and a fixed nonlinearity; real neurons are vastly more complex, and how brains learn is not settled. The engineering stands on its own.

Misconception

"Initialising every weight to zero is a safe neutral start." It makes every unit in a layer identical forever. Random initialisation at a scale matched to the activation is required.

Exercises

Exercises

1. Neural networks and backpropagation

Warm-upHuman

A fully connected network has 4 inputs, one hidden layer of 8 units, and 3 outputs, with a bias for every unit. How many parameters does it have?

2. Neural networks and backpropagation

Warm-upHuman

A student builds a 10-layer network but forgets the activation functions, so every layer is just Wx+bW\mathbf{x} + \mathbf{b}. What can the network represent?

Choose one

3. Neural networks and backpropagation

CoreHuman

Let f=(xy+z)2f = (xy + z)^2 with x=2x = 2, y=3y = 3, z=−4z = -4. Using the chain rule backwards through the graph, compute ∂f/∂x\partial f / \partial x.

4. Neural networks and backpropagation

CoreComputational

The starter contains a scalar Var class in the style of the lesson: each operation stores its inputs together with the local derivatives. The backward method is missing. Implement it:

  1. Collect every Var the output depends on, in a topological order (each node after all of its inputs).
  2. Set every grad to 0, then the output's grad to 1.
  3. Visit the nodes in reverse order; for each node v and each (parent, local) in v.inputs, add local * v.grad to parent.grad.

A value used twice (like a in a * a) must receive the sum of both contributions.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

5. Neural networks and backpropagation

StretchComputational

Implement the backward pass for a one-hidden-layer network on a batch, as derived in the lesson but with a tanh hidden layer and squared error:

Z1=XW1+b1,A1=tanh⁡Z1,y^=A1W2+b2,L=1n∑i(y^i−yi)2.Z_1 = XW_1 + \mathbf{b}_1,\quad A_1 = \tanh Z_1,\quad \hat{\mathbf{y}} = A_1 W_2 + b_2,\quad L = \frac{1}{n}\sum_i (\hat y_i - y_i)^2 .

grads(X, y, W1, b1, W2, b2) must return a dict with keys "W1", "b1", "W2", "b2" holding ∂L\partial L with respect to each, with the same shapes as the parameters. Shapes: X is (n, d), y is (n,), W1 is (d, h), b1 is (h,), W2 is (h,), b2 is a float.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

6. Neural networks and backpropagation

CoreHuman

A two-layer tanh network has every weight and bias initialised to exactly 0. It is trained by gradient descent. What happens?

Choose one

7. Neural networks and backpropagation

CoreHuman

Which statements follow from the universal approximation theorem as stated in the lesson? Select all that apply.

Choose all that apply

8. Neural networks and backpropagation

CoreComputational

For multi-class output layers you need the softmax and the cross-entropy of the true class. Write softmax_xent(z, label) taking a 1-D numpy array of scores z and the index of the true class, returning (probs, loss) where probs is the softmax of z and loss is −log⁡(probs[label])-\log(\text{probs}[\text{label}]).

It must work for huge scores such as [1000, 1001, 1002] without overflow. (Hint: softmax is unchanged if you subtract the same constant from every score; and log⁡softmax(z)k=zk−log⁡∑jezj\log \text{softmax}(\mathbf{z})_k = z_k - \log\sum_j e^{z_j}.)

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources