Beyond straight lines
The models of Learning from data are linear: logistic regression separates classes with a straight line (or flat hyperplane). Many real boundaries are curved. The simplest example is XOR: label a point 1 if exactly one of its two binary coordinates is 1. The points and are positive, and negative, and no single line separates them.
One fix is to engineer features by hand — add , or — and keep the model linear in them. That works when you know which features matter. For images, sound or text, nobody does. Neural networks learn the features too: they compose many simple, adjustable units so that early layers compute useful features and later layers combine them. This lesson builds them from the ground up, together with backpropagation, the algorithm that computes their gradients efficiently and makes training them possible at all.
Neurons and layers
Definition— Neuron (unit)
A function of an input vector with weights , a bias and a fixed nonlinear activation function : . The quantity is its pre-activation.
Logistic regression is a single neuron with . A layer of neurons applied to the same input computes , with a weight matrix and applied elementwise. A multilayer perceptron (MLP), or fully connected feedforward network, composes layers:
The intermediate are hidden layers; the number of units in each is its width, the number of layers the network's depth. The last layer is usually linear, followed by a sigmoid or softmax inside the loss for classification. A layer from inputs to outputs has parameters, so a 784-128-10 network for digit images has of them.
The nonlinearity is essential:
Theorem— Without activations, depth adds nothing
A composition of affine maps is an affine map. So a network whose layers all use the identity activation computes a function of the form , no matter how many layers it has.
Proof
For two layers, , which is affine with matrix and bias . Induction on the number of layers finishes the proof. ∎
∎
Activation functions
| Activation | Formula | Derivative | Notes |
|---|---|---|---|
| Sigmoid | outputs in ; saturates for large | ||
| Tanh | zero-centred, outputs in ; also saturates | ||
| ReLU | if , else | cheap; no saturation for ; units can "die" | |
| GELU | a smooth ReLU; , are the standard normal CDF and density |
Sigmoid and tanh were the classic choices, but both saturate: for large their derivative is nearly zero, and since backpropagation multiplies derivatives layer by layer, gradients in deep sigmoid networks shrink towards nothing — the vanishing gradient problem. The sigmoid's derivative never exceeds , so ten sigmoid layers can scale a gradient by , about one millionth, even at best.
The rectified linear unit (ReLU) changed deep learning in the early 2010s: its derivative is exactly 1 wherever it is active, so gradients pass through unchanged. Its flaw is that a unit whose pre-activation is negative for every input has zero gradient and never recovers (a dead unit). GELU weights its input by the probability that a standard normal variable is below it; it behaves like ReLU for large but is smooth near zero. GELU was used in BERT and GPT-2, and smooth, gated relatives (such as the SwiGLU unit) are common in today's large language models. In practice the choice among the modern options matters less than initialisation, normalisation and the learning rate — but the choice of some nonlinearity is essential.
What one hidden layer can do: universal approximation
How expressive are these networks? The classic answer is a theorem proved for sigmoids by George Cybenko (1989) and Kurt Hornik (1991), and in a general form by Leshno, Lin, Pinkus and Schocken (1993).
Theorem— Universal approximation (one hidden layer)
Let be continuous and not a polynomial. Let be compact (closed and bounded), continuous, and . Then there exist a number of units and parameters , such that
The proof needs functional analysis (the Hahn–Banach and Riesz representation theorems in Cybenko's version), but in one dimension the idea is visible with ReLUs: each unit adds a hinge that changes the slope by at the point , so a sum of hinges can follow any continuous curve as closely as you like, piece by linear piece. The block below sets the weights by hand to interpolate a wiggly function and measures the worst error as units are added.
import numpy as np
# One hidden layer of ReLU units, with weights set by hand rather than learned:
# each unit adds a "hinge" that changes the slope at one point, so the network
# is the piecewise-linear interpolation of the target at evenly spaced knots.
def relu(z):
return np.maximum(z, 0)
def hinge_network(f, a, b, units):
knots = np.linspace(a, b, units + 1)
values = f(knots)
slopes = np.diff(values) / np.diff(knots)
# output(x) = values[0] + sum_k c_k * relu(x - knots[k]), with c_k the change in slope at knot k
c = np.concatenate([[slopes[0]], np.diff(slopes)])
def net(x):
hidden = relu(x[:, None] - knots[None, :-1]) # one column per hidden unit
return values[0] + hidden @ c # a linear output layer
return net
target = lambda x: np.sin(3 * x) + 0.5 * x
x = np.linspace(-2, 2, 2001)
print("hidden units worst error on [-2, 2]")
for units in (2, 4, 8, 16, 32, 64, 128):
net = hinge_network(target, -2, 2, units)
print(f"{units:12d} {np.max(np.abs(net(x) - target(x))):.5f}")Each doubling of the width cuts the error by about four, as you would expect from piecewise-linear interpolation of a smooth function. Now be careful about what the theorem does not say:
- It says nothing about size. can be astronomically large; for some functions in dimensions, shallow networks need a number of units exponential in . Depth helps: Telgarsky (2016) constructed functions that deep ReLU networks represent with few units but shallow ones need exponentially many to approximate.
- It says nothing about learning. The weights exist; the theorem does not say gradient descent finds them.
- It says nothing about generalisation. Approximating a function well on all of is different from learning it from finitely many noisy examples, which is the problem of the previous lesson.
So "neural networks can represent anything" is true only in this limited sense. Why they learn so well in practice is a deeper and still partly open question.
Computational graphs
To train a network by gradient descent we need for every parameter — possibly billions of them. The key is to see the computation as a graph.
A computational graph is a directed acyclic graph whose nodes are values and whose edges say which values were computed from which. Inputs and parameters are leaves; each other node is an elementary operation (add, multiply, tanh, …) applied to its parents; the loss is the final node. Here is the graph of a one-example squared error for a single linear unit, , with the numbers , , , .
The forward pass evaluates the nodes from left to right. The backward pass goes right to left, computing for every node its adjoint .
The chain rule and backpropagation
In one variable, the chain rule says that if depends on and on , then . In a graph, a value can influence the loss along several paths, and the multivariable chain rule adds them up: for a node whose value is used by children ,
Each is a local derivative, known from the single operation that computed : for it is ; for it is 1; for it is . Backpropagation is just this formula applied to every node in reverse topological order, so that every child's adjoint is ready before it is needed. Start with .
Example— Backpropagation by hand
In the figure: . Since , . Since , . Since , and . Since , (and , which we would not need unless were a parameter). Gradient descent would therefore increase and , pushing the prediction up towards .
Why backwards? Forward-mode differentiation pushes derivatives from one input towards the outputs, so it costs one pass per input — a billion passes for a billion parameters. Reverse mode pulls derivatives from one output back to all inputs at once, so one backward pass gives the whole gradient of a scalar loss, at a cost of a small constant multiple of the forward pass. That asymmetry is why backpropagation made large networks trainable. Its price is memory: the backward pass needs the intermediate values of the forward pass, so training must store activations — the main reason training needs far more memory than running a model.
Backpropagation, then, is not a learning rule: it is an efficient way to compute gradients. Gradient descent (or the optimisers of the next lesson) does the learning.
A tiny automatic-differentiation engine
Frameworks such as PyTorch and JAX implement reverse mode for arrays. The essential idea fits in a page for scalars. In the design below, every operation computes its result and its local derivatives during the forward pass and stores them with the result; backward then only has to sort the graph and apply the sum-over-children formula. The block checks the engine against the hand calculation and against finite differences, then uses it to train a tiny network on XOR.
import math
import random
class Var:
"""A scalar that remembers how it was computed. Each Var stores, for every input it was
computed from, the local derivative d(self)/d(input), worked out during the forward pass."""
def __init__(self, value, inputs=()):
self.value = value
self.inputs = inputs # tuple of (input Var, local derivative)
self.grad = 0.0
# Each operation computes its value AND its local derivatives, then wraps them.
def __add__(self, other):
other = other if isinstance(other, Var) else Var(other)
return Var(self.value + other.value, ((self, 1.0), (other, 1.0)))
def __mul__(self, other):
other = other if isinstance(other, Var) else Var(other)
return Var(self.value * other.value, ((self, other.value), (other, self.value)))
def __neg__(self):
return self * -1.0
def __sub__(self, other):
return self + (-other if isinstance(other, Var) else Var(-other))
def __pow__(self, k): # k is a plain number
return Var(self.value ** k, ((self, k * self.value ** (k - 1)),))
__radd__ = __add__
__rmul__ = __mul__
def tanh(self):
t = math.tanh(self.value)
return Var(t, ((self, 1 - t * t),))
def relu(self):
return Var(max(self.value, 0.0), ((self, 1.0 if self.value > 0 else 0.0),))
def exp(self):
e = math.exp(self.value)
return Var(e, ((self, e),))
def log(self):
return Var(math.log(self.value), ((self, 1 / self.value),))
def backward(self):
"""Reverse-mode differentiation: fill .grad = d(self)/d(v) for every v this depends on."""
order, seen = [], set()
def visit(v): # topological order: inputs before the values built from them
if id(v) not in seen:
seen.add(id(v))
for parent, _ in v.inputs:
visit(parent)
order.append(v)
visit(self)
for v in order:
v.grad = 0.0
self.grad = 1.0
for v in reversed(order): # outputs first: each node passes its gradient back
for parent, local in v.inputs:
parent.grad += local * v.grad # chain rule, summed over every path
# 1. The worked example from the text: L = (w*x + b - y)^2
w, b = Var(0.5), Var(-1.0)
x, y = 3.0, 2.0
L = (w * x + b - y) ** 2
L.backward()
print(f"L = {L.value}, dL/dw = {w.grad}, dL/db = {b.grad}")
# 2. Check against finite differences on a messier expression.
def f(a, c):
return ((a * c).tanh() + (a ** 2) * c.exp()).log()
a, c = Var(0.7), Var(-0.3)
out = f(a, c)
out.backward()
h = 1e-6
num_a = (f(Var(0.7 + h), Var(-0.3)).value - f(Var(0.7 - h), Var(-0.3)).value) / (2 * h)
num_c = (f(Var(0.7), Var(-0.3 + h)).value - f(Var(0.7), Var(-0.3 - h)).value) / (2 * h)
print(f"autograd: {a.grad:.8f} {c.grad:.8f}")
print(f"numeric: {num_a:.8f} {num_c:.8f}")
# 3. Train a 2-3-1 network on XOR, which no linear model can fit.
random.seed(1)
W1 = [[Var(random.uniform(-1, 1)) for _ in range(2)] for _ in range(3)]
b1 = [Var(0.0) for _ in range(3)]
W2 = [Var(random.uniform(-1, 1)) for _ in range(3)]
b2 = Var(0.0)
params = [p for row in W1 for p in row] + b1 + W2 + [b2]
def predict(x1, x2):
hidden = [(W1[j][0] * x1 + W1[j][1] * x2 + b1[j]).tanh() for j in range(3)]
return sum((W2[j] * hidden[j] for j in range(3)), b2)
data = [((0, 0), 0), ((0, 1), 1), ((1, 0), 1), ((1, 1), 0)]
for step in range(301):
loss = sum(((predict(*xs) - t) ** 2 for xs, t in data), Var(0.0)) * 0.25
loss.backward()
for p in params:
p.value -= 0.2 * p.grad
if step % 100 == 0:
print(f"step {step:3d}: loss {loss.value:.4f}")
print("XOR predictions:", [round(predict(*xs).value, 2) for xs, _ in data])About fifty lines of Python, and the network learns XOR — something no linear model can do. The structure is exactly that of an industrial framework: a graph recorded during the forward pass, local derivatives per operation, a reverse sweep accumulating adjoints. What frameworks add is operations on whole tensors (so one node is a matrix multiply, not one multiplication), GPU kernels, and many engineering refinements.
Backpropagation for a whole layer, in numpy
A scalar engine creates one node per multiplication, far too slow for real networks. For an MLP we can apply the chain rule to whole matrices once, by hand. Take a batch of examples as the rows of , one hidden ReLU layer and a sigmoid output with cross-entropy:
Going backwards, with meaning and elementwise multiplication:
Each line is the scalar rule in matrix form, and a dimension check catches most mistakes: must have the shape of , and does. The first line is the same "prediction minus truth" you met in logistic regression. The code checks the formulas numerically before training a network on the "two moons" dataset, then draws the boundary it learned.
import numpy as np
rng = np.random.default_rng(0)
def moons(n, noise=0.15):
"""Two interleaving half-circles: a classic dataset no straight line can separate."""
t = rng.uniform(0, np.pi, n)
upper = np.column_stack([np.cos(t), np.sin(t)])
lower = np.column_stack([1 - np.cos(t), 0.5 - np.sin(t)])
X = np.vstack([upper, lower]) + rng.normal(0, noise, (2 * n, 2))
y = np.concatenate([np.zeros(n), np.ones(n)])
return X, y
X, y = moons(200)
X_test, y_test = moons(200)
# A 2 -> 32 -> 1 network: ReLU hidden layer, sigmoid output, binary cross-entropy loss.
H = 32
params = {
"W1": rng.normal(0, np.sqrt(2 / 2), (2, H)), # He initialisation (see below)
"b1": np.zeros(H),
"W2": rng.normal(0, np.sqrt(1 / H), (H, 1)),
"b2": np.zeros(1),
}
def forward(p, X):
Z1 = X @ p["W1"] + p["b1"]
A1 = np.maximum(Z1, 0)
Z2 = A1 @ p["W2"] + p["b2"]
return Z1, A1, Z2[:, 0]
def loss_and_grads(p, X, y):
n = len(y)
Z1, A1, z = forward(p, X)
loss = np.mean(np.logaddexp(0, -z) * y + np.logaddexp(0, z) * (1 - y))
# Backward pass: the chain rule, layer by layer, from the loss back to the inputs.
dz = (0.5 * (1 + np.tanh(z / 2)) - y) / n # dL/dz for sigmoid + cross-entropy
grads = {"W2": A1.T @ dz[:, None], "b2": np.array([dz.sum()])}
dA1 = dz[:, None] @ p["W2"].T
dZ1 = dA1 * (Z1 > 0) # ReLU passes gradient only where active
grads["W1"] = X.T @ dZ1
grads["b1"] = dZ1.sum(axis=0)
return loss, grads
# Check the hand-derived gradients against finite differences before trusting them.
loss, grads = loss_and_grads(params, X, y)
for name, idx in [("W1", (1, 3)), ("b1", (5,)), ("W2", (7, 0)), ("b2", (0,))]:
old = params[name][idx]
params[name][idx] = old + 1e-5; up = loss_and_grads(params, X, y)[0]
params[name][idx] = old - 1e-5; down = loss_and_grads(params, X, y)[0]
params[name][idx] = old
print(f"d loss / d {name}{list(idx)}: backprop {grads[name][idx]: .7f} numeric {(up - down) / 2e-5: .7f}")
def accuracy(p, X, y):
return np.mean((forward(p, X)[2] > 0) == y)
for step in range(3001):
loss, grads = loss_and_grads(params, X, y)
for name in params:
params[name] -= 0.5 * grads[name]
if step % 1000 == 0:
print(f"step {step:4d}: loss {loss:.4f}, train accuracy {accuracy(params, X, y):.1%}")
print(f"test accuracy: {accuracy(params, X_test, y_test):.1%}")
# The learned decision regions, drawn in text: '#' where the network predicts class 1.
for row in np.linspace(1.4, -0.9, 11):
cols = np.linspace(-1.4, 2.4, 48)
grid = np.column_stack([cols, np.full_like(cols, row)])
print("".join("#" if z > 0 else "." for z in forward(params, grid)[2]))The network bends its boundary between the two moons, which no linear classifier could do. Note the habit on display: check hand-written gradients numerically before trusting them. A wrong gradient often still decreases the loss a little, so a bug can hide for a long time.
Initialisation
Where training starts matters more than you might expect.
Break the symmetry. If every weight starts at the same value (zero, say), every hidden unit in a layer computes the same function, receives the same gradient, and stays identical forever: a hundred units behave as one. Weights must start random. Biases can start at zero.
Get the scale right. Consider one unit with independent zero-mean weights of variance , independent of the inputs. Then . If is above the "right" value, the signal grows layer after layer; below it, the signal shrinks — exponentially in depth either way, and the gradients do the same on the way back. For tanh near zero, which is roughly linear, keeping the scale steady needs ; Glorot and Bengio (2010) proposed to balance the forward and backward passes. ReLU zeroes half of its inputs on average, halving , so He and colleagues (2015) doubled it: .
import numpy as np
rng = np.random.default_rng(0)
width, depth = 256, 20
x = rng.normal(0, 1, (1000, width)) # a batch of standardised inputs
def activation_spread(act, weight_std):
"""Push the batch through `depth` layers; report the std of activations at some layers."""
h, report = x, {}
for layer in range(1, depth + 1):
W = rng.normal(0, weight_std, (width, width))
h = act(h @ W)
if layer in (1, 5, 10, 20):
report[layer] = h.std()
return report
relu = lambda z: np.maximum(z, 0)
settings = [
("ReLU, std 0.01", relu, 0.01),
("ReLU, std 1 / sqrt(fan_in)", relu, np.sqrt(1 / width)),
("ReLU, He: sqrt(2 / fan_in)", relu, np.sqrt(2 / width)),
("ReLU, std 0.1", relu, 0.1),
("tanh, std 0.01", np.tanh, 0.01),
("tanh, Xavier: sqrt(1 / fan_in)", np.tanh, np.sqrt(1 / width)),
]
print(f"{'initialisation':32s}" + "".join(f"{'layer ' + str(k):>12s}" for k in (1, 5, 10, 20)))
for name, act, std in settings:
r = activation_spread(act, std)
print(f"{name:32s}" + "".join(f"{r[k]:12.3g}" for k in (1, 5, 10, 20)))With He initialisation, the activations of a 20-layer ReLU network keep roughly the same size all the way down. Too small, and they vanish within a few layers; slightly too large, and they grow without bound. These rules assume particular activations and layer types, and modern networks also use normalisation layers and residual connections (next lessons) that make training much less sensitive to the exact scale. But a bad initialisation can still stop a network from learning at all, and frameworks choose these defaults for you for good reason.
A short history
Frank Rosenblatt's perceptron (1958) was a single trainable neuron with a threshold activation. Marvin Minsky and Seymour Papert's book Perceptrons (1969) analysed the limits of single-layer networks, XOR among them, and interest waned. Reverse-mode differentiation was described by Seppo Linnainmaa in 1970 and proposed for neural networks by Paul Werbos in 1974; the 1986 paper by Rumelhart, Hinton and Williams showed convincingly that backpropagation lets multilayer networks learn useful internal representations, and it spread widely. Yann LeCun's convolutional networks read handwritten postcodes around 1990 and bank cheques later that decade. Networks then went out of fashion again until GPUs, large labelled datasets and techniques like ReLU made very deep networks trainable; the turning point is usually dated to 2012, when the AlexNet network won the ImageNet image-recognition competition by a wide margin. The rest of this level, and the next, follow from there.
Common misconceptions
Misconception
"The universal approximation theorem explains why deep learning works." It says a large enough network can represent any continuous function on a compact set. It says nothing about how large, whether training finds the weights, or whether the result generalises from data.
Misconception
"Backpropagation is how neural networks learn." Backpropagation computes gradients. An optimiser such as gradient descent uses them to change the parameters. They are separate pieces, and the same backpropagation serves many optimisers.
Misconception
"Artificial neurons work like biological neurons." The name is historical inspiration. A unit is a weighted sum and a fixed nonlinearity; real neurons are vastly more complex, and how brains learn is not settled. The engineering stands on its own.
Misconception
"Initialising every weight to zero is a safe neutral start." It makes every unit in a layer identical forever. Random initialisation at a scale matched to the activation is required.
Exercises
1. Neural networks and backpropagation
Warm-upHumanA fully connected network has 4 inputs, one hidden layer of 8 units, and 3 outputs, with a bias for every unit. How many parameters does it have?
2. Neural networks and backpropagation
Warm-upHumanA student builds a 10-layer network but forgets the activation functions, so every layer is just . What can the network represent?
3. Neural networks and backpropagation
CoreHumanLet with , , . Using the chain rule backwards through the graph, compute .
4. Neural networks and backpropagation
CoreComputationalThe starter contains a scalar Var class in the style of the lesson: each operation stores its inputs together with the local derivatives. The backward method is missing. Implement it:
- Collect every
Varthe output depends on, in a topological order (each node after all of its inputs). - Set every
gradto 0, then the output'sgradto 1. - Visit the nodes in reverse order; for each node
vand each(parent, local)inv.inputs, addlocal * v.gradtoparent.grad.
A value used twice (like a in a * a) must receive the sum of both contributions.
5. Neural networks and backpropagation
StretchComputationalImplement the backward pass for a one-hidden-layer network on a batch, as derived in the lesson but with a tanh hidden layer and squared error:
grads(X, y, W1, b1, W2, b2) must return a dict with keys "W1", "b1", "W2", "b2" holding with respect to each, with the same shapes as the parameters. Shapes: X is (n, d), y is (n,), W1 is (d, h), b1 is (h,), W2 is (h,), b2 is a float.
6. Neural networks and backpropagation
CoreHumanA two-layer tanh network has every weight and bias initialised to exactly 0. It is trained by gradient descent. What happens?
7. Neural networks and backpropagation
CoreHumanWhich statements follow from the universal approximation theorem as stated in the lesson? Select all that apply.
8. Neural networks and backpropagation
CoreComputationalFor multi-class output layers you need the softmax and the cross-entropy of the true class. Write softmax_xent(z, label) taking a 1-D numpy array of scores z and the index of the true class, returning (probs, loss) where probs is the softmax of z and loss is .
It must work for huge scores such as [1000, 1001, 1002] without overflow. (Hint: softmax is unchanged if you subtract the same constant from every score; and .)
Further reading and resources
- Learning representations by back-propagating errors · David Rumelhart, Geoffrey Hinton and Ronald Williams · 1986The Nature paper that made backpropagation widely known.
- Approximation by superpositions of a sigmoidal function · George Cybenko · 1989
- Multilayer feedforward networks with a nonpolynomial activation function can approximate any function · Moshe Leshno, Vladimir Lin, Allan Pinkus and Shimon Schocken · 1993
- Benefits of depth in neural networks · Matus Telgarsky · 2016
- Automatic Differentiation in Machine Learning: a Survey · Baydin, Pearlmutter, Radul and Siskind · 2018
- Understanding the difficulty of training deep feedforward neural networks · Xavier Glorot and Yoshua Bengio · 2010
- Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification · Kaiming He et al. · 2015Introduces the initialisation now called He initialisation.
- Gaussian Error Linear Units (GELUs) · Dan Hendrycks and Kevin Gimpel · 2016
- CS231n: Backpropagation, intuitionsStanford course notes with many small worked examples of gradients flowing through circuits.
- micrograd · Andrej KarpathyA well-known tiny autograd engine for teaching; compare its design with the one in this lesson.
- Deep Learning · Ian Goodfellow, Yoshua Bengio and Aaron CourvilleFree online. Chapter 6 covers feedforward networks and backpropagation.