When the order of the data matters
A grid operator must decide, a day ahead, how much generation to schedule for every hour. Schedule too little and the frequency sags; schedule too much and expensive plants burn fuel for nothing. The operator's main tool is a forecast built from years of past demand, and the past demand is a time series: measurements of one quantity, taken in order, at regular intervals.
Time series are everywhere a quantity is tracked: monthly unemployment, daily hospital admissions, weekly sales of school uniforms, the carbon dioxide concentration measured at Mauna Loa since 1958, river levels, sunspot counts. They differ from the data of most earlier lessons in one decisive way. Earlier, we treated observations as independent draws, so shuffling the rows changed nothing. In a time series, order is information. This month's demand is close to last month's; a cold January is followed by a fairly cold February. That dependence is what makes forecasting possible, and it is also what breaks methods that assume independence: standard errors come out too small, two unrelated trending series appear strongly related, and a model tested on randomly chosen rows quietly peeks at the future.
This lesson covers the classic toolkit: decomposing a series into trend, seasonality and noise; measuring dependence with autocorrelation; the idea of stationarity; the AR, MA and ARIMA models of Box and Jenkins; exponential smoothing; and the part most often done badly, evaluating forecasts honestly.
Trend, seasonality and noise
Most series that people forecast have three visible ingredients. The trend is the slow movement of the level: demand growing with population, a disease declining with vaccination. The seasonal component repeats with a fixed period: electricity peaks in winter in Warsaw and in summer in Riyadh; retail sales peak before festivals; hospital admissions follow the week. The remainder is whatever is left. An additive decomposition writes and suits series whose seasonal swings stay the same size as the level changes. When the swings grow in proportion to the level, as airline passenger numbers or sales of a growing product often do, a multiplicative decomposition fits better, and taking logarithms turns it into an additive one.
The classical way to estimate the parts is simple. A moving average spanning exactly one season smooths the seasonal pattern away and leaves the trend. Subtracting the trend leaves seasonality plus noise, and averaging that by calendar month gives the seasonal indices. The code below does this for eight years of synthetic monthly electricity demand for a mid-sized city, built so that we know the true components.
import numpy as np
rng = np.random.default_rng(42)
n = 96 # eight years of months, January 2017 onwards
t = np.arange(n)
trend = 300 + 1.2 * t # GWh per month, slowly growing
season = 30 * np.cos(2 * np.pi * t / 12) + 12 * np.cos(4 * np.pi * t / 12) # winter peak, small summer bump
noise = np.zeros(n)
for i in range(1, n): # AR(1) noise: today's surprise lingers
noise[i] = 0.5 * noise[i - 1] + rng.normal(0, 8)
y = trend + season + noise
# Classical additive decomposition.
# 1. Trend: a centred 2x12 moving average (weights 1/24, 1/12, ..., 1/12, 1/24).
w = np.r_[0.5, np.ones(11), 0.5] / 12
trend_hat = np.full(n, np.nan)
trend_hat[6:-6] = np.convolve(y, w, mode="valid")
# 2. Seasonal indices: average detrended value for each calendar month, centred to sum to zero.
detrended = y - trend_hat
idx = np.array([np.nanmean(detrended[m::12]) for m in range(12)])
idx -= idx.mean()
# 3. Remainder.
remainder = y - trend_hat - np.tile(idx, n // 12)
months = "Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec".split()
print("month estimated index true seasonal effect")
for m in range(12):
print(f" {months[m]} {idx[m]:+6.1f} {season[m]:+6.1f}")
r = remainder[~np.isnan(remainder)]
print(f"remainder: {r.size} months, standard deviation {r.std():.1f} GWh")
def acf(x, max_lag):
x = x - x.mean()
return np.array([np.sum(x[k:] * x[:len(x) - k]) / np.sum(x * x) for k in range(1, max_lag + 1)])
print("autocorrelation of the remainder, lags 1-6:", np.round(acf(r, 6), 2),
f" (white-noise band: +/-{1.96 / np.sqrt(r.size):.2f})")The estimated indices track the truth to within a few GWh: January is 43.8 above the trend (truth 42.0), September 22.0 below (truth 21.0). Two details deserve attention. The moving average needs six months on each side, so the trend is missing for the first and last half-year, exactly where a forecaster most wants it; production methods handle the ends with forecasts or local regression. And the remainder is not pure noise: its lag-1 autocorrelation of 0.38 is outside the white-noise band of ±0.21, because we built the noise so that each month's surprise carries into the next. That leftover dependence is what the models later in the lesson capture.
Autocorrelation and stationarity
Definition— autocorrelation function
For a series with mean , the sample autocorrelation at lag is the correlation of the series with itself shifted by steps. Plotted against it is the autocorrelation function (ACF), or correlogram.
If the series is white noise (independent, identically distributed values), each is approximately normal with mean 0 and standard deviation , so about 95% of lags fall within . Spikes outside the band are evidence of dependence, though with 20 lags plotted, one spike outside the band is expected by chance alone.
Definition— weak stationarity
A series is (weakly) stationary if its mean and variance are the same at every time, and its autocovariance depends only on the lag , not on .
Stationarity is what lets us learn from the past: if the rules of dependence stayed the same, last decade's data describe next year. A series with a trend or a seasonal pattern is not stationary (its mean changes with time), and neither is a random walk, , whose variance grows without bound. A random walk can be made stationary by differencing, ; seasonal patterns by seasonal differencing, . A series that becomes stationary after differences is said to be integrated of order . Formal unit-root tests (the augmented Dickey–Fuller and KPSS tests) help decide how many differences are needed.
import numpy as np
rng = np.random.default_rng(1974)
def acf(x, max_lag):
x = x - x.mean()
return np.array([np.sum(x[k:] * x[:len(x) - k]) / np.sum(x * x) for k in range(1, max_lag + 1)])
n = 200
e = rng.normal(0, 1, n)
walk = np.cumsum(e) # random walk: y_t = y_{t-1} + e_t
ar = np.zeros(n)
for i in range(1, n):
ar[i] = 0.7 * ar[i - 1] + e[i] # stationary AR(1)
print("lags: 1 2 3 4 5 10")
for name, x in [("white noise", e), ("AR(1), phi = 0.7", ar), ("random walk", walk), ("differenced walk", np.diff(walk))]:
r = acf(x, 10)
print(f"{name:18}" + "".join(f"{v:6.2f}" for v in r[[0, 1, 2, 3, 4, 9]]))
# Spurious regression: regress one random walk on another, completely independent one.
def slope_t(x, y):
X = np.column_stack([np.ones(len(x)), x])
coef, *_ = np.linalg.lstsq(X, y, rcond=None)
resid = y - X @ coef
s2 = resid @ resid / (len(y) - 2)
return coef[1] / np.sqrt(s2 * np.linalg.inv(X.T @ X)[1, 1])
reps = 2000
t_levels, t_diffs = np.empty(reps), np.empty(reps)
for r in range(reps):
a, b = np.cumsum(rng.normal(size=n)), np.cumsum(rng.normal(size=n))
t_levels[r], t_diffs[r] = slope_t(a, b), slope_t(np.diff(a), np.diff(b))
print(f"independent random walks: 'significant' slope (|t| > 1.97) in {np.mean(np.abs(t_levels) > 1.97):.0%} of {reps} regressions")
print(f"their differences: 'significant' slope in {np.mean(np.abs(t_diffs) > 1.97):.0%}")The second half of the program is one of the most important experiments in applied statistics. Two random walks generated independently have no relationship, yet regressing one on the other gives a "significant" slope at the 5% level in 83% of 2,000 tries. Clive Granger and Paul Newbold called this spurious regression in 1974. The ordinary standard errors assume independent residuals; with two wandering series, the residuals wander too, and the effective sample size is far smaller than 200. Regress the differences instead and the false-positive rate returns to the advertised 5%. This is why a correlation between two trending national statistics, however strong, is weak evidence of anything.
AR, MA and ARIMA models
Definition— autoregressive and moving-average models
With white noise of variance :
- an autoregressive model of order , AR(), is : today is a weighted sum of recent values plus a fresh shock;
- a moving-average model of order , MA(), is : today is a weighted sum of recent shocks;
- ARMA() combines both, and ARIMA() is an ARMA() model for the series differenced times.
Theorem— The AR(1) process
If with , the stationary process has mean , variance and autocorrelations .
Proof
Taking expectations of both sides with a constant mean gives . For the variance, depends only on shocks up to time , so it is uncorrelated with , and . For , multiply the centred equation by and take expectations; is uncorrelated with the past, so . Hence and . If the variance equation has no positive solution: the process wanders or explodes.
∎
So an AR process has an ACF that decays geometrically (or as a damped wave), while an MA() process, which only remembers shocks, has an ACF that cuts off after lag . The mirror-image tool is the partial autocorrelation function (PACF), the correlation at lag after removing the effect of lags , which cuts off after lag for an AR(). George Box and Gwilym Jenkins built a whole method (identify, estimate, check) around reading these two plots, in their 1970 book. Today the order is usually chosen by an information criterion, AIC or BIC, from Likelihood and model selection.
An AR model is a regression of the series on its own past, so it can be fitted by least squares. Below, an AR(2) with and generates a series with a pseudo-cycle: its characteristic roots are complex, so shocks echo as damped oscillations. Udny Yule fitted exactly this kind of model to sunspot numbers in 1927, the first autoregression.
import numpy as np
rng = np.random.default_rng(1927)
n = 300
y = np.zeros(n)
for t in range(2, n): # AR(2) with a pseudo-cycle
y[t] = 10 + 1.3 * y[t - 1] - 0.6 * y[t - 2] + rng.normal(0, 2)
y = y[50:] # drop the start-up stretch
n = len(y)
def fit_ar(y, p, start):
"""Least-squares AR(p): regress y_t on 1, y_{t-1}, ..., y_{t-p}, using t = start, ..., n-1."""
X = np.column_stack([np.ones(len(y) - start)] + [y[start - k:len(y) - k] for k in range(1, p + 1)])
target = y[start:]
coef, *_ = np.linalg.lstsq(X, target, rcond=None)
resid = target - X @ coef
return coef, resid, X
# Choose the order by AIC, fitting every candidate on the same stretch of data.
print(" p AIC")
for p in range(1, 6):
coef, resid, X = fit_ar(y, p, start=5)
m = len(resid)
print(f" {p} {m * np.log(resid @ resid / m) + 2 * (p + 1):7.1f}")
coef, resid, X = fit_ar(y, 2, start=2)
s2 = resid @ resid / (len(resid) - 3)
se = np.sqrt(np.diag(s2 * np.linalg.inv(X.T @ X)))
print("AR(2) estimates (true 10, 1.3, -0.6):", np.round(coef, 3), " standard errors:", np.round(se, 3))
c, phi1, phi2 = coef
roots = np.roots([1, -phi1, -phi2]) # z^2 - phi1 z - phi2 = 0
print(f"root modulus {abs(roots[0]):.3f} (< 1, so stationary); implied cycle {2 * np.pi / abs(np.angle(roots[0])):.1f} steps")
print(f"long-run mean {c / (1 - phi1 - phi2):.2f}, observed mean {y.mean():.2f}, residual sd {np.sqrt(s2):.2f}")
# Forecast 12 steps ahead by feeding forecasts back in.
hist = list(y[-2:])
for h in range(12):
hist.append(c + phi1 * hist[-1] + phi2 * hist[-2])
print("last two values:", np.round(y[-2:], 1), " forecasts:", np.round(hist[2:], 1))AIC drops sharply from order 1 (458.5) to order 2 (351.8) and then creeps up, so it picks the true order. The estimates, 1.240 and −0.594, are within 1.2 standard errors of the truth. The implied cycle of 9.9 steps is close to the true 10.9, and the forecasts show what every stationary model does in the long run: starting from 30.2, they overshoot slightly to 34.0 and then settle at the long-run mean of about 33. The further ahead we look, the less the past tells us, and the forecast becomes the mean.
Exponential smoothing
A different tradition, born in operational research in the 1950s with Robert Brown's and Charles Holt's work on inventory forecasting, starts from a simple idea: forecast with a weighted average of past observations, giving recent ones more weight. Simple exponential smoothing keeps a level and updates it after each observation: with . Unrolled, the forecast is : weights decaying geometrically into the past. A large reacts fast and is noisy; a small one is smooth and slow.
Holt's method adds a smoothed slope, so forecasts continue a trend, and Holt–Winters (Peter Winters, 1960) adds a smoothed seasonal index for each position in the season. These methods were long regarded as useful heuristics, until Rob Hyndman and colleagues showed around 2002 that each corresponds to a statistical state-space model (the ETS family, for error, trend and seasonal), with a likelihood, information criteria and proper prediction intervals. Some coincide with ARIMA models; simple exponential smoothing, for instance, gives the optimal forecasts for an ARIMA(0, 1, 1). The two traditions are now two views of the same territory, and good software offers both.
Judging forecasts honestly
A model's fit to the data it was estimated on says little about its forecasts; with enough parameters, any series can be fitted closely. The honest test imitates real use. Pick a forecast origin, fit using only data up to that point, forecast the next value(s), record the error, move the origin forward one step and repeat. This rolling-origin evaluation (also called time-series cross-validation) never lets a model see the future it is judged on. Randomly splitting months into training and test sets, as in ordinary cross-validation, would let a model "forecast" March 2020 from February and April 2020 on either side.
Errors need a scale. The mean absolute error (MAE) is in the data's units; the root mean squared error (RMSE) penalises large misses more; and the mean absolute scaled error (MASE) divides the MAE by the in-sample MAE of a simple benchmark, so that MASE below 1 means "better than the benchmark", whatever the series' units. Percentage errors (MAPE) are popular but break down when values are near zero and treat over- and under-forecasts unequally.
Every evaluation should include benchmarks: the naive forecast (next month equals this month), the seasonal naive (next January equals this January), and perhaps the mean. Sophisticated methods fail to beat them more often than one would hope. The code below runs a rolling-origin comparison on the electricity series: 36 one-month-ahead forecasts, re-tuning each method's smoothing parameters at every origin using only the data available then.
import itertools
import numpy as np
# The same synthetic electricity series as before: eight years of monthly demand in GWh.
rng = np.random.default_rng(42)
n = 96
t = np.arange(n)
noise = np.zeros(n)
for i in range(1, n):
noise[i] = 0.5 * noise[i - 1] + rng.normal(0, 8)
y = 300 + 1.2 * t + 30 * np.cos(2 * np.pi * t / 12) + 12 * np.cos(4 * np.pi * t / 12) + noise
def ses_one_step(x, alpha):
"""Simple exponential smoothing: one-step forecasts for x[1:], and the forecast after the end."""
level, f = x[0], []
for v in x[1:]:
f.append(level)
level += alpha * (v - level)
return np.array(f), level
def hw_one_step(x, a, b, g, m=12):
"""Additive Holt-Winters. Initialised from the first two seasons; returns one-step forecasts
for x[m:], and the forecast after the end."""
level = x[:m].mean()
slope = (x[m:2 * m].mean() - x[:m].mean()) / m
season = list(x[:m] - level)
f = []
for i in range(m, len(x)):
s = season[i - m]
f.append(level + slope + s)
new_level = a * (x[i] - s) + (1 - a) * (level + slope)
slope = b * (new_level - level) + (1 - b) * slope
season.append(g * (x[i] - new_level) + (1 - g) * s)
level = new_level
return np.array(f), level + slope + season[len(x) - m]
def best(fn, x, grid, skip):
"""Smoothing parameters that minimise the in-sample one-step squared error."""
return min(grid, key=lambda p: np.sum((fn(x, *p)[0] - x[skip:]) ** 2))
methods = {"naive (last month)": [], "seasonal naive": [], "exponential smoothing": [], "Holt-Winters": []}
hw_grid = list(itertools.product([0.1, 0.3, 0.5], [0.01, 0.1], [0.1, 0.3, 0.5]))
for origin in range(60, n): # rolling origin: train on y[:origin], forecast y[origin]
train, actual = y[:origin], y[origin]
alpha = best(ses_one_step, train, [(a,) for a in np.arange(0.05, 1.0, 0.05)], 1)
params = best(hw_one_step, train, hw_grid, 12)
methods["naive (last month)"].append(train[-1] - actual)
methods["seasonal naive"].append(train[-12] - actual)
methods["exponential smoothing"].append(ses_one_step(train, *alpha)[1] - actual)
methods["Holt-Winters"].append(hw_one_step(train, *params)[1] - actual)
scale = np.mean(np.abs(y[12:60] - y[:48])) # in-sample MAE of the seasonal naive method
print(f"{len(methods['naive (last month)'])} one-month-ahead forecasts, origins from month 60 to 95")
print("method MAE RMSE MASE")
for name, err in methods.items():
err = np.array(err)
print(f"{name:22} {np.mean(np.abs(err)):6.1f} {np.sqrt(np.mean(err ** 2)):6.1f} {np.mean(np.abs(err)) / scale:5.2f}")Holt–Winters wins clearly, with an MAE of 6.4 GWh against 11.0 for the best benchmark, and a MASE of 0.41. The ranking of the benchmarks is instructive too. The seasonal naive forecast, which one might expect to do well on a seasonal series, is the worst here (MAE 14.2), because copying last January ignores a whole year of growth: about GWh. Simple exponential smoothing barely beats the naive forecast, since it models neither trend nor season. A method that captures all three components does best, as it should, because the simulation was built from exactly those components.
The large forecasting competitions organised by Spyros Makridakis and colleagues turned such comparisons into a science. In the M4 competition (100,000 series), the most accurate entries combined several statistical methods or blended them with machine learning, and simple benchmarks were hard to beat; in M5 (hierarchical retail sales), gradient-boosted trees trained across many related series did best. The durable lessons: always compare against benchmarks, evaluate out of sample, and report uncertainty, not just point forecasts.
Structural breaks
Every method above assumes the future resembles the past. Sometimes the rules change: a new tariff changes electricity demand, a change in a survey's questionnaire shifts the unemployment rate, a lockdown in 2020 changes hospital admissions, traffic and retail sales within days. A structural break is a change in the level, trend, seasonal pattern or dependence of a series at some point in time.
Example— Detecting a break
If the date is known in advance (the day a law took effect), Gregory Chow's 1960 test compares a model fitted separately before and after with one fitted to the whole series, using an statistic. If the date is unknown, search over candidate dates for the one that most reduces the residual sum of squares. Because the search picks the best of many dates, its test statistic must be compared with a distribution that accounts for the search, not with the ordinary table; Jushan Bai and Pierre Perron's 1998 method extends this to several breaks. The exercises ask you to write the search for a single shift in the mean.
What to do about a break depends on its cause. A one-off level shift can be modelled with an indicator variable; a lasting change may mean that only data since the break are relevant, at the cost of less data; and a temporary disruption, such as the pandemic months, is often modelled as an intervention so that it does not distort the seasonal pattern estimated for normal years. Statistical agencies faced exactly this choice in 2020 when seasonally adjusting their headline series.
Time series in professional practice
National statistical offices publish seasonally adjusted series using programs such as the US Census Bureau's X-13ARIMA-SEATS and the European Statistical System's JDemetra+, which combine ARIMA models, outlier detection and calendar effects (Easter moves, months have different numbers of working days). Central banks, energy system operators and retailers forecast thousands of series automatically. In Python, statsmodels provides ARIMA, exponential smoothing and unit-root tests; in R, the fable family accompanies Hyndman and Athanasopoulos's free textbook. The snippet below (not run here; statsmodels is not loaded in the browser) fits a seasonal ARIMA and a Holt–Winters model to a monthly series.
# Python with statsmodels, not run here
from statsmodels.tsa.arima.model import ARIMA
from statsmodels.tsa.holtwinters import ExponentialSmoothing
sarima = ARIMA(y, order=(1, 0, 0), seasonal_order=(0, 1, 1, 12), trend="t").fit()
hw = ExponentialSmoothing(y, trend="add", seasonal="add", seasonal_periods=12).fit()
print(sarima.forecast(12), hw.forecast(12))Since about 2023, research groups have released pretrained foundation models for time series, transformer networks trained on very large collections of series and applied to new ones without re-training: examples include Google Research's TimesFM and Amazon's Chronos. As of 2026 the evidence on them is mixed and active: on some benchmarks they match tuned statistical methods with no tuning at all, on others well-tuned exponential smoothing or gradient boosting is as good or better, and their performance after structural breaks is a live question. The evaluation discipline of this lesson, rolling origins, honest benchmarks and calibrated intervals, is exactly what separates real progress from leaderboard noise. Probabilistic forecasts, judged with the methods of Conformal prediction and proper scoring rules, are increasingly the standard.
Common misconceptions
Misconception
"Ordinary cross-validation works for time series." Shuffling observations into folds lets the model learn from the future and from neighbours of the very points it is tested on, which are almost copies of them. Errors come out optimistic, sometimes dramatically. Use rolling-origin evaluation, with a gap between training and test data if dependence is long.
Misconception
"A strong correlation between two series over time shows they are linked." Any two series that both trend, up or down, will be strongly correlated. Before interpreting the association, difference or detrend the series, or test for cointegration.
Misconception
"A stationary series has no pattern." Stationarity means the rules of dependence do not change, not that there is no dependence. The AR(2) series above has a clear ten-step cycle and is stationary; a random walk with no pattern at all to the eye is not.
Misconception
"The model with the best in-sample fit forecasts best." In-sample fit always improves with more parameters. Forecast accuracy usually peaks at a modest level of complexity, and combinations of simple methods are often among the most accurate forecasts available.
Exercises
1. Time series
Warm-upHumanWhich of these series is (weakly) stationary?
2. Time series
Warm-upHumanDaily deviations of a reservoir's water level from its seasonal norm follow an AR(1) model with . What is the autocorrelation between deviations three days apart, as a percentage rounded to the nearest whole number?
3. Time series
CoreHumanA pharmacy in Accra forecasts weekly sales of a cough syrup by simple exponential smoothing with . The current level is 100 bottles. The next three weeks' sales are 140, 100 and 130. What is the forecast for the week after, in bottles?
4. Time series
CoreHumanA stationary series of 400 observations has a sample ACF with one large spike at lag 1 () and all other lags inside the band . Its partial autocorrelations decay gradually, alternating in sign. Which model is the natural first candidate?
5. Time series
CoreHumanA team forecasting daily hospital admissions in Manila splits five years of days at random into 80% training and 20% test data, fits a model and reports an excellent test error. Why should a reviewer distrust the result?
6. Time series
CoreComputationalWrite fit_ar1(y), which fits the model by least squares, regressing on a constant and . Return the pair (c, phi) as Python floats.
For example, fit_ar1([1, 2, 3, 4, 5]) returns (1.0, 1.0), since each value is 1 plus the previous one.
7. Time series
StretchComputationalA sensor's readings may have shifted to a new mean at some unknown time. Write best_break(y, min_size=5), which returns the index k that minimises the total sum of squared deviations when the series is split into y[:k] and y[k:], each segment around its own mean. Both segments must have at least min_size values. Return k as a Python int.
For example, best_break([0.0] * 10 + [5.0] * 10) is 10.
Further reading and resources
- Forecasting: Principles and Practice (3rd edition) · Rob J. Hyndman and George AthanasopoulosA free online textbook covering decomposition, exponential smoothing, ARIMA and forecast evaluation, with R code.
- Spurious regressions in econometrics · Clive W. J. Granger and Paul Newbold · 1974
- On a method of investigating periodicities in disturbed series, with special reference to Wolfer's sunspot numbers · G. Udny Yule · 1927The first autoregressive model.
- A state space framework for automatic forecasting using exponential smoothing methods · Rob J. Hyndman, Anne B. Koehler, Ralph D. Snyder and Simone Grose · 2002
- Another look at measures of forecast accuracy · Rob J. Hyndman and Anne B. Koehler · 2006Introduces the mean absolute scaled error (MASE).
- Out-of-sample tests of forecasting accuracy: an analysis and review · Leonard J. Tashman · 2000Rolling-origin evaluation and its pitfalls.
- The M4 Competition: 100,000 time series and 61 forecasting methods · Spyros Makridakis, Evangelos Spiliotis and Vassilios Assimakopoulos · 2020
- M5 accuracy competition: results, findings, and conclusions · Spyros Makridakis, Evangelos Spiliotis and Vassilios Assimakopoulos · 2022
- Tests of equality between sets of coefficients in two linear regressions · Gregory C. Chow · 1960
- Estimating and testing linear models with multiple structural changes · Jushan Bai and Pierre Perron · 1998
- Trends in atmospheric carbon dioxide (Mauna Loa and global) · NOAA Global Monitoring LaboratoryA public-domain monthly series with a clear trend and seasonal cycle, good for practice.
- X-13ARIMA-SEATS seasonal adjustment program · United States Census Bureau
- statsmodels: time series analysis (tsa)