When the formula runs out
A water charity in Malawi records how many days it takes to repair each broken village hand pump. For the last 24 repairs the mean is 5.8 days, and the board asks how precise that figure is. For a mean there is a formula, , and a interval to go with it — though both lean on a normal approximation that is shaky for 24 strongly skewed values. But the board's next questions have no textbook formula at all. How precise is the median repair time? The 90th percentile, which says how long the unluckiest villages wait? The ratio of this year's mean to last year's?
Each of these questions asks the same thing: how much would my estimate change if I could repeat the data collection? Classical statistics answers it by mathematics, deriving the sampling distribution of each statistic, often only approximately and only for some statistics. In 1979 Bradley Efron proposed answering it by computation instead: if we cannot draw new samples from the population, draw them from the best picture of the population we have — the sample itself. That idea, the bootstrap, is one of a family of resampling methods that replace derivations with repeated computation on the data:
- the bootstrap estimates the uncertainty of almost any estimate;
- permutation tests test hypotheses by reshuffling labels;
- cross-validation estimates how well a prediction method will do on new data;
- Monte Carlo studies test the methods themselves, on simulated data where the truth is known.
You have already used two of them: the percentile bootstrap in Estimation and confidence intervals and a permutation test in Comparing groups. This lesson takes each method in depth — why it works, its better variants, and how to test it — and then spends as much care on when it fails.
The bootstrap world
The variation of an estimate around the true comes from sampling: a different sample from the population would give a different . We cannot see that variation, because we have one sample. The bootstrap builds a parallel world in which we can.
In the bootstrap world the "population" is the empirical distribution , which puts probability on each observed value, and its "true parameter" is , which we know. Drawing a sample of size from means drawing values from the data with replacement: some observations appear twice or more, others not at all. On average a resample contains of the distinct original observations. Computing the statistic on many resamples shows how varies around , and we use that as a stand-in for how varies around . This substitution of for is the plug-in principle you met with the percentile interval; for smooth statistics and reasonably large samples, theory confirms that it works.
Bootstrap estimates and intervals
Definition— nonparametric bootstrap
Given data and an estimate , draw resamples , each of values chosen with replacement from the data, and compute the bootstrap replicates , . The bootstrap standard error is the standard deviation of the replicates, and the bootstrap estimate of bias is .
Three kinds of 95% interval are built from the replicates, with denoting the -quantile of the :
- Percentile interval: . Simple, and it respects the range of the parameter (a bootstrap interval for a proportion never leaves ).
- Basic interval: . It assumes that is distributed like and inverts that relationship; for skewed statistics it is reflected the opposite way from the percentile interval.
- BCa interval (bias-corrected and accelerated): the percentile interval with its quantile levels adjusted. A bias correction measures how far the replicates sit off-centre, and an acceleration , estimated from the jackknife (the estimates obtained by leaving out one observation at a time), measures how fast the standard error changes with the parameter, which is what makes sampling distributions skewed. The interval is with where is the standard normal cdf and its quantiles. With it reduces to the percentile interval.
BCa endpoints are second-order accurate: under regularity conditions the coverage error of a one-sided BCa bound shrinks like , against for percentile and basic bounds. That is a statement about large samples; the simulation study below tests what it means for 24 observations.
The bootstrap at work
The program resamples the 24 repair times 10,000 times, computes the bootstrap standard error and bias, the three bootstrap intervals and the interval, and implements BCa by hand before checking it against scipy.stats.bootstrap.
import numpy as np
from scipy import stats
# Synthetic data: days taken to repair 24 broken village hand pumps.
days = np.array([3.3, 5.5, 7.3, 2.1, 8.3, 5.5, 8.4, 5.6, 11.4, 2.1, 18.6, 11.7,
2.8, 7.6, 6.4, 1.1, 7.2, 2.8, 3.7, 2.8, 3.2, 4.7, 4.8, 1.3])
n, B = len(days), 10_000
rng = np.random.default_rng(1979)
theta_hat = days.mean()
idx = rng.integers(0, n, size=(B, n)) # B resamples of size n, with replacement
boot = days[idx].mean(axis=1) # the statistic on every resample
se = boot.std(ddof=1)
print(f"sample mean {theta_hat:.3f} days; bootstrap SE {se:.3f}; formula s/sqrt(n) {days.std(ddof=1) / np.sqrt(n):.3f}")
print(f"bootstrap estimate of bias: {boot.mean() - theta_hat:+.3f}")
lo, hi = np.quantile(boot, [0.025, 0.975])
print(f"percentile interval [{lo:.2f}, {hi:.2f}]")
print(f"basic interval [{2 * theta_hat - hi:.2f}, {2 * theta_hat - lo:.2f}]")
t = stats.t.ppf(0.975, n - 1) * days.std(ddof=1) / np.sqrt(n)
print(f"t interval [{theta_hat - t:.2f}, {theta_hat + t:.2f}]")
# BCa: correct the percentile interval for bias (z0) and for skewness (acceleration a).
z0 = stats.norm.ppf(np.mean(boot < theta_hat))
jack = np.array([np.delete(days, i).mean() for i in range(n)]) # leave-one-out estimates
d = jack.mean() - jack
a = (d**3).sum() / (6 * ((d**2).sum()) ** 1.5)
z = stats.norm.ppf([0.025, 0.975])
adj = stats.norm.cdf(z0 + (z0 + z) / (1 - a * (z0 + z)))
lo_bca, hi_bca = np.quantile(boot, adj)
print(f"BCa interval [{lo_bca:.2f}, {hi_bca:.2f}] (z0 = {z0:.3f}, a = {a:.3f}, quantiles used {adj.round(4)})")
res = stats.bootstrap((days,), np.mean, n_resamples=B, method="BCa", random_state=np.random.default_rng(1))
print(f"scipy.stats.bootstrap BCa [{res.confidence_interval.low:.2f}, {res.confidence_interval.high:.2f}]")The bootstrap standard error, 0.799 days, is slightly smaller than the formula's 0.816. That is not an error: the bootstrap world's population has variance , with divisor rather than , so it reproduces . The bias estimate, −0.009 days, is pure noise, as it should be for a sample mean, which is unbiased.
The intervals are more interesting. The repair times are skewed to the right — one pump took 18.6 days — so the sampling distribution of their mean is skewed to the right too, and an honest interval should reach further above 5.76 than below it. The interval is symmetric by construction, and the basic interval leans the wrong way. The BCa interval, , leans the right way: its positive and acceleration move both quantile levels up, to 4.8% and 99.0%. SciPy's BCa interval agrees to within a few hundredths, the size of the Monte Carlo variation between two sets of 10,000 resamples.
Permutation tests
Comparing groups built a permutation test by shuffling labels thousands of times. Here we ask why that works, and how to do it exactly. An agronomy station in Ethiopia randomly assigns 8 of 16 maize plots to a cover-crop treatment, and measures yields in tonnes per hectare (the data are synthetic). The cover-crop plots average 0.50 t/ha more. Could that be chance?
If the cover crop does nothing at all, each plot's yield would have been the same whichever group it was assigned to; the labels "cover crop" and "control" are then arbitrary, and every way of choosing 8 of the 16 plots was equally likely to be the one we got. So we can compute the difference in means for every possible assignment and ask how unusual the real one is. This is the logic R. A. Fisher gave in The Design of Experiments (1935): the randomisation that was actually performed is the basis of the test.
Theorem— Permutation tests are exact under exchangeability
Suppose that under the null hypothesis the joint distribution of the data is unchanged by permuting the group labels (the data are exchangeable). Let be the observed test statistic and its values under all relabellings other than the observed one, and define the p-value as the fraction of all values that are at least as extreme as . Then for every , under the null hypothesis.
Proof
Under exchangeability, the observed labelling is, in distribution, just one of the equally likely labellings, so the rank of among is uniformly distributed (with ties broken against rejection). The p-value is at most only when ranks among the most extreme values, which has probability at most .
∎
When there are too many relabellings to enumerate, we draw of them at random. The same argument holds if we count the observed labelling among them and use . Belinda Phipson and Gordon Smyth showed in 2010 that omitting the "+1", which many programs once did, understates p-values and can report the impossible value zero.
import itertools
import numpy as np
from scipy import stats
# Synthetic field trial: maize yield (tonnes per hectare) on 16 plots, 8 randomly given a cover crop.
control = np.array([3.1, 2.8, 3.5, 2.6, 3.0, 3.3, 2.9, 2.4])
cover = np.array([3.4, 3.9, 3.2, 3.6, 2.9, 4.1, 3.5, 3.0])
pooled = np.concatenate([cover, control])
observed = cover.mean() - control.mean()
print(f"observed difference in means: {observed:.3f} t/ha")
# Exact: every way of choosing which 8 of the 16 plots were "cover crop" (16 choose 8 = 12,870).
total = pooled.sum()
diffs = np.array([pooled[list(c)].sum() / 8 - (total - pooled[list(c)].sum()) / 8
for c in itertools.combinations(range(16), 8)])
p_exact = np.mean(np.abs(diffs) >= abs(observed) - 1e-12) # the 1e-12 allows for rounding error
print(f"exact two-sided permutation p-value: {p_exact:.4f} ({np.sum(np.abs(diffs) >= abs(observed) - 1e-12)} of {len(diffs)} splits)")
# Monte Carlo: 9,999 random relabellings, counting the observed labelling as one of them.
rng = np.random.default_rng(1935)
B = 9_999
perm = np.array([rng.permutation(pooled) for _ in range(B)])
mc = perm[:, :8].mean(axis=1) - perm[:, 8:].mean(axis=1)
p_mc = (1 + np.sum(np.abs(mc) >= abs(observed) - 1e-12)) / (B + 1)
print(f"Monte Carlo permutation p-value: {p_mc:.4f}")
res = stats.permutation_test((cover, control), lambda a, b: a.mean() - b.mean(),
permutation_type="independent", n_resamples=np.inf)
print(f"scipy.stats.permutation_test (exact): {res.pvalue:.4f}")
print(f"Welch t-test: {stats.ttest_ind(cover, control, equal_var=False).pvalue:.4f}")All splits can be enumerated in under a second: 352 of them give a difference at least as large as 0.50 in either direction, so the exact p-value is 0.027. The random-permutation estimate, SciPy's exact computation and Welch's test (0.023) all tell the same story. When they agree, the permutation test's value is reassurance; when they disagree, usually because of small samples or outliers, the permutation test is the one whose assumptions are easier to defend — if the data are exchangeable under the null.
Cross-validation is resampling too
The bootstrap asks how much an estimate varies; cross-validation asks how well a prediction method will do on data it has not seen. In -fold cross-validation, the data are split into roughly equal folds; each fold in turn is held out, the method is fitted to the other , and its errors on the held-out fold are recorded. The average of these errors estimates the method's prediction error. Stephen Bates, Trevor Hastie and Robert Tibshirani have shown that it estimates the average error of the fitting procedure over training sets like ours, rather than the error of the particular model we end up with, and that the usual standard errors for it are too small.
Cross-validation is honest only if the held-out fold plays no part in any step of the fitting. The classic trap is to select variables using all the data and then cross-validate only the final model. The program reproduces that trap on pure noise: 50 cases, 2,000 random features, labels unrelated to any of them, and a simple nearest-centroid classifier using the 20 features most correlated with the labels. Every classifier has a true error rate of 50%.
import numpy as np
rng = np.random.default_rng(2002)
n, p, keep, K = 50, 2_000, 20, 5
def nearest_centroid_error(Xtr, ytr, Xte, yte):
"""Classify each test row by the closer of the two class means in the training data."""
c0, c1 = Xtr[ytr == 0].mean(axis=0), Xtr[ytr == 1].mean(axis=0)
pred = (((Xte - c1) ** 2).sum(axis=1) < ((Xte - c0) ** 2).sum(axis=1)).astype(int)
return np.mean(pred != yte)
def top_features(X, y, k):
"""The k features most correlated (in absolute value) with the labels."""
Xc, yc = X - X.mean(axis=0), y - y.mean()
corr = (Xc * yc[:, None]).sum(axis=0) / np.sqrt((Xc**2).sum(axis=0) * (yc**2).sum())
return np.argsort(-np.abs(corr))[:k]
wrong, right = [], []
for rep in range(20):
X = rng.normal(size=(n, p)) # pure noise: no feature carries any signal
y = rng.permutation(np.repeat([0, 1], n // 2)) # labels unrelated to X
folds = np.arange(n) % K
rng.shuffle(folds)
chosen_once = top_features(X, y, keep) # WRONG: select using all the data
e_wrong, e_right = [], []
for k in range(K):
tr, te = folds != k, folds == k
e_wrong.append(nearest_centroid_error(X[tr][:, chosen_once], y[tr], X[te][:, chosen_once], y[te]))
chosen = top_features(X[tr], y[tr], keep) # RIGHT: select inside each training fold
e_right.append(nearest_centroid_error(X[tr][:, chosen], y[tr], X[te][:, chosen], y[te]))
wrong.append(np.mean(e_wrong)); right.append(np.mean(e_right))
print("estimated error rate when the true error rate of any classifier is 0.50")
print(f" selection outside cross-validation: mean {np.mean(wrong):.3f} (range {min(wrong):.2f} to {max(wrong):.2f})")
print(f" selection inside cross-validation: mean {np.mean(right):.3f} (range {min(right):.2f} to {max(right):.2f})")Selecting the features outside the cross-validation produces an estimated error of 4%: a "discovery" of near-perfect prediction from noise, in every one of the 20 simulated data sets. Moving the selection inside each fold gives 50%, the truth. The leak is that the held-out cases helped choose the features, so they are no longer new to the model. The same leak occurs with scaling, imputation of missing values, oversampling, and tuning of hyperparameters; in software, a pipeline object that bundles every step and is refitted inside each fold is the standard defence (see the Data Science elective's machine learning for tabular data). Notice also how much the honest estimate varies — from 0.34 to 0.68 with 50 cases — another reason to report cross-validated errors with humility.
Monte Carlo studies: testing methods where the truth is known
How do we know whether a method — an interval, a test, a sampler — works? We cannot check it on real data, where the truth is unknown. But we can on simulated data, where we chose the truth. A Monte Carlo study (or simulation study) generates many data sets from a known model, applies the method to each, and measures its performance: bias, coverage, error rates, power. Tim Morris, Ian White and Michael Crowther's guide organises a study by its aims, data-generating mechanisms, estimands, methods and performance measures (ADEMP), and asks that the Monte Carlo standard error of every performance measure be reported.
The program runs two such studies. The first draws 1,000 samples of 24 values from a lognormal population shaped like the repair times, whose true mean is known, and checks how often each 95% interval covers it. The second tests a Bayesian computation by simulation-based calibration (SBC), proposed by Sean Talts and colleagues in 2018: draw a "true" parameter from the prior, simulate data from it, run the inference, and record the rank of the true value among the posterior draws. If the computation is correct, the true value is just another draw from the posterior, so its rank is uniformly distributed. We test the correct beta–binomial posterior and a buggy one that counts every observation twice.
import numpy as np
from scipy import stats
rng = np.random.default_rng(2019)
# Study 1: how often do 95% intervals for a mean cover the truth, for skewed data?
# Population: lognormal with log-mean 1.5 and log-sd 0.8 (like the repair times); true mean known.
mu, sigma, n, reps, B = 1.5, 0.8, 24, 1_000, 1_000
true_mean = np.exp(mu + sigma**2 / 2)
z = stats.norm.ppf([0.025, 0.975])
hits = {"t": 0, "percentile": 0, "BCa": 0}
above = {"t": 0, "percentile": 0, "BCa": 0} # misses with the whole interval below the truth
widths = {"t": 0.0, "percentile": 0.0, "BCa": 0.0}
for _ in range(reps):
x = rng.lognormal(mu, sigma, n)
m = x.mean()
boot = x[rng.integers(0, n, size=(B, n))].mean(axis=1)
h = stats.t.ppf(0.975, n - 1) * x.std(ddof=1) / np.sqrt(n)
jack = (x.sum() - x) / (n - 1)
d = jack.mean() - jack
a = (d**3).sum() / (6 * ((d**2).sum()) ** 1.5)
z0 = stats.norm.ppf(np.mean(boot < m))
ivs = {"t": (m - h, m + h),
"percentile": tuple(np.quantile(boot, [0.025, 0.975])),
"BCa": tuple(np.quantile(boot, stats.norm.cdf(z0 + (z0 + z) / (1 - a * (z0 + z)))))}
for k, (lo, hi) in ivs.items():
hits[k] += lo <= true_mean <= hi
above[k] += hi < true_mean
widths[k] += hi - lo
print(f"Coverage of nominal 95% intervals for a lognormal mean, n = {n}, {reps} simulated data sets")
for k in hits:
print(f" {k:10} coverage {hits[k] / reps:.3f} average width {widths[k] / reps:.2f}"
f" misses: interval too low {above[k]}, too high {reps - hits[k] - above[k]}")
print(f" (Monte Carlo standard error of each coverage: about {np.sqrt(0.95 * 0.05 / reps):.3f})")
# Study 2: simulation-based calibration of a Bayesian computation (beta-binomial, n = 20).
def sbc_ranks(posterior, sims=2_000, draws=99):
ranks = np.empty(sims, dtype=int)
for s in range(sims):
theta = rng.beta(2, 2) # 1. draw a "true" parameter from the prior
k = rng.binomial(20, theta) # 2. simulate data from it
post = posterior(k, draws) # 3. run the inference being tested
ranks[s] = np.sum(post < theta) # 4. rank of the truth among the posterior draws
return ranks
correct = lambda k, L: rng.beta(2 + k, 2 + 20 - k, L)
buggy = lambda k, L: rng.beta(2 + 2 * k, 2 + 2 * (20 - k), L) # counts every observation twice
for name, post in [("correct posterior", correct), ("overconfident bug", buggy)]:
counts = np.bincount(sbc_ranks(post) // 10, minlength=10) # ranks 0..99 in 10 bins
p = stats.chisquare(counts).pvalue
print(f"\n{name}: rank counts in 10 bins (200 expected in each), chi-square p = {p:.3g}")
for i, c in enumerate(counts):
print(f" ranks {10 * i:2}-{10 * i + 9:2} {c:4} " + "#" * (c // 10))The first study is humbling. With 24 skewed observations, no method delivers 95%: all three cover about 90% of the time, well outside the Monte Carlo error of 0.007. The direction of the misses is telling: the interval missed 95 times by lying entirely below the truth and only 3 times above it, because samples that miss the rare large values have both small means and small standard deviations. BCa balances its misses better (71 and 28), as its design intends, but it cannot conjure up information about a tail that the sample did not see. The practical lesson is that "second-order accurate" is a promise about large samples, and for skewed data with small a Monte Carlo study like this one, tailored to your situation, is worth running before you trust any interval.
The second study shows what SBC can detect. The correct posterior gives a flat histogram ( for a chi-square test of uniformity). The buggy posterior, too narrow because it treats 20 observations as 40, leaves the true value outside the bulk of its draws far too often: the histogram is U-shaped, with . Other shapes have other meanings: a hump in the middle means the posterior is too wide, and a slope means it is biased in one direction. SBC is now a standard check for new samplers and for complex models fitted with Stan or PyMC, because it tests the whole pipeline — model code, sampler and summaries — at once.
When resampling fails
Resampling methods feel assumption-free. They are not, and their assumptions are worth knowing as precisely as a formula's.
- Extremes. For the sample maximum, the bootstrap fails outright. A resample includes the largest observation with probability , so about 63% of the bootstrap replicates equal the sample maximum exactly. For a population with an upper limit, such as a uniform distribution, the true sampling distribution of the maximum is continuous and lies entirely below that limit; the bootstrap distribution, a spike at the sample maximum with scattered values below it, looks nothing like it. The same problem affects other statistics that depend on a few extreme values, and estimates at the boundary of the parameter space.
- Heavy tails. If the population has infinite variance, the bootstrap distribution of the mean does not converge to the right limit. With finite but large variance it merely converges slowly, as the coverage study showed.
- Dependence. Resampling individual observations assumes they are independent. For time series it destroys the autocorrelation and understates the uncertainty; the block bootstrap resamples whole stretches of consecutive observations instead. For clustered data — pupils in schools, measurements on the same patient — resample whole clusters. Cross-validation needs the same care: folds that split a patient's records, or put tomorrow's data in the training set and yesterday's in the test set, leak information.
- Non-exchangeability. A permutation test of equal means assumes that, under the null, the groups are interchangeable. If they have different variances and different sizes, they are not, and the test can reject a true null too often or too rarely. Permuting a studentised statistic, such as Welch's , restores approximate validity.
- The data themselves. Resampling a biased sample reproduces its bias exactly. No amount of computation repairs a sample that left out the people, places or times that matter.
Resampling in practice
History. Maurice Quenouille's jackknife (1949), named by John Tukey, was the first resampling method for bias and variance. Efron's 1979 bootstrap generalised it and became practical as computers spread; his 1987 BCa interval and the books by Efron and Robert Tibshirani (1993) and by Anthony Davison and David Hinkley (1997) set out the theory. Permutation tests go back to Fisher and to Edwin Pitman in the 1930s, when enumerating relabellings was done by hand for tiny samples. Cross-validation was formalised by Mervyn Stone and Seymour Geisser in the 1970s, and is today the default way to compare prediction methods.
Tools. In Python, scipy.stats has bootstrap (percentile, basic and BCa), permutation_test and monte_carlo_test, and scikit-learn has cross-validation splitters (including GroupKFold and TimeSeriesSplit) and pipelines. In R, the classic boot package does the same, shown here for comparison (not run here):
# R, not run here: bootstrap intervals for the mean repair time with the boot package
library(boot)
b <- boot(days, function(x, i) mean(x[i]), R = 10000)
boot.ci(b, type = c("perc", "basic", "bca"))Habits. Resample in the same way the data were generated: individuals if they were sampled independently, clusters if clusters were sampled, blocks for time series, and within strata for stratified designs. Use thousands of resamples and record the seed. Keep every data-driven step inside cross-validation. Report Monte Carlo error. And when a method is new to you or your data are unusual, run a small simulation study before trusting it — it is the statistician's equivalent of testing code.
Common misconceptions
Misconception
"The bootstrap creates new data." It reuses the data you have, to estimate how an estimate would vary. It cannot increase the information in the sample, and it cannot reveal values the sample never contained.
Misconception
"Bootstrap intervals are exact, or always better than formulas." They are approximations whose accuracy improves with . For 24 skewed values, the percentile and BCa intervals covered about 90% of the time, like the interval. For the sample maximum the bootstrap is simply wrong.
Misconception
"A permutation test makes no assumptions." It assumes exchangeability under the null hypothesis, and it tests a specific null — usually that the groups' distributions are identical, not merely that their means are equal.
Misconception
"The cross-validated error is the error of my final model." It estimates the average error of the fitting procedure on training sets of that size, and it is honest only if every step of the procedure, including variable selection and tuning, was repeated inside each fold.
Misconception
"None of 10,000 random permutations was as extreme as the data, so ." The observed labelling is itself one of the possible labellings, so the p-value is at least . Report , or better, run more permutations if the exact size matters.
Exercises
1. Resampling and simulation
Warm-upHumanThe data are . Which of these could be a bootstrap resample of them?
2. Resampling and simulation
CoreHumanA data set has distinct observations. On average, how many distinct observations does a bootstrap resample of size 1,000 contain? Round to the nearest whole number.
3. Resampling and simulation
Warm-upHumanTwelve tomato plants are randomly split into two groups of 6, one fed with a new compost. For an exact permutation test, how many distinct ways are there to choose which 6 plants form the compost group?
4. Resampling and simulation
CoreHumanAn analyst has 120 patients' gene-expression profiles (20,000 genes each) and wants to estimate how accurately a classifier predicts response to a treatment. Which procedure gives an honest estimate of its error rate?
5. Resampling and simulation
CoreHumanIn which of these situations would the ordinary nonparametric bootstrap (resampling individual observations with replacement) give misleading uncertainty? Select all that apply.
6. Resampling and simulation
CoreComputationalWrite bootstrap_median_ci(x, B=5_000, seed=0) that returns the 95% percentile bootstrap interval for the median of x as a pair (low, high). Draw B resamples of len(x) values with replacement using np.random.default_rng(seed), take the median of each, and return the 2.5% and 97.5% quantiles of those medians.
The tests use the 24 repair times from the lesson and a large normal sample, for which the interval's width should be close to the normal-theory value .
7. Resampling and simulation
StretchComputationalWrite permutation_pvalue(a, b, B=9_999, seed=0) that returns a two-sided Monte Carlo permutation p-value for the difference in means between groups a and b. Shuffle the pooled data B times with np.random.default_rng(seed), split each shuffle into groups of the original sizes, and count the shuffles whose absolute difference in means is at least the observed one (allowing 1e-12 for rounding error). Count the observed labelling too: return (count + 1) / (B + 1).
The tests use the cover-crop trial from the lesson (exact p-value 0.0274), perfectly separated groups, and identical groups.
Further reading and resources
- Bootstrap Methods: Another Look at the Jackknife · Bradley Efron · 1979The paper that introduced the bootstrap.
- Better Bootstrap Confidence Intervals · Bradley Efron · 1987The BCa interval, with discussion.
- Permutation P-values Should Never Be Zero · Belinda Phipson and Gordon Smyth · 2010Why Monte Carlo permutation p-values should count the observed labelling.
- Cross-validation: what does it estimate and how well does it do it? · Stephen Bates, Trevor Hastie and Robert Tibshirani · 2021
- Using simulation studies to evaluate statistical methods · Tim Morris, Ian White and Michael Crowther · 2019A practical guide to planning, running and reporting Monte Carlo studies.
- Validating Bayesian Inference Algorithms with Simulation-Based Calibration · Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari and Andrew Gelman · 2018
- SciPy: scipy.stats.bootstrapPercentile, basic and BCa intervals; the same module has permutation_test and monte_carlo_test.
- scikit-learn: cross-validationK-fold, grouped and time-series splitters, and pipelines that keep preprocessing inside the folds.