Twenty chances to be lucky
A team evaluating a wellbeing app for university students in Lyon randomises 60 volunteers to the app or a waiting list and, to be thorough, measures twenty outcomes: sleep, stress, mood, concentration, exercise, and fifteen more. One of the twenty differs between the groups with . The press release writes itself: app reduces evening anxiety.
Suppose the app does nothing at all. Each test still has a 5% chance of giving , and with twenty tests the chance that at least one does is . A "significant" result somewhere among twenty is not surprising evidence; it is the expected outcome of looking twenty times. The 0.05 threshold controls the error rate of one test, and the team ran twenty.
The same problem appears at every scale. A genome-wide association study tests around a million genetic variants for association with a disease, so geneticists require , which is divided by a million. A particle physicist scanning a range of masses for a bump in the data must allow for the many places a bump could have appeared, which physicists call the look-elsewhere effect, and is one reason they demand five standard deviations before announcing a discovery. And — most dangerously, because it is invisible — an analyst who tries several reasonable analyses of one data set and reports the one that worked has also tested many hypotheses, even if the paper shows only one test.
This lesson covers both halves of the problem: the formal methods for correcting many explicit tests, and the wider question of why many published findings have failed to replicate, and what the scientific community is doing about it.
What chance does with many tries
The key fact, which you met in Hypothesis testing, is that when a null hypothesis is true and the test statistic is continuous, the p-value is uniformly distributed between 0 and 1. (If has continuous distribution function under the null and large values count against it, the p-value is , and for every — the probability integral transform from Lumenary Math's Probability theory.) So under the null, happens 5% of the time, happens 1% of the time, and a p-value near zero is exactly as likely as one near 0.5. When there is a real effect, p-values pile up near zero instead, and that pile-up is what tests detect.
The program simulates studies comparing two groups of 30 with a test, first when every null is true, for families of 1 to 50 tests, and then shows the distribution of p-values with and without a real effect.
import numpy as np
from scipy import stats
rng = np.random.default_rng(2005)
reps = 10_000
# Each "study" compares two groups of 30 with a t-test. Here no treatment works: every null is true.
def null_pvalues(m, chunk=1_000):
out = []
for _ in range(reps // chunk): # in chunks, to keep memory small
a = rng.normal(size=(chunk, m, 30))
b = rng.normal(size=(chunk, m, 30))
out.append(stats.ttest_ind(a, b, axis=2).pvalue)
return np.concatenate(out) # shape (reps, m)
print(" tests P(at least one p < 0.05) theory 1 - 0.95^m average number of 'discoveries'")
for m in [1, 5, 10, 20, 50]:
p = null_pvalues(m)
print(f"{m:6} {np.mean((p < 0.05).any(axis=1)):17.3f} {1 - 0.95**m:20.3f} {np.mean((p < 0.05).sum(axis=1)):22.2f}")
# Under a true null, p-values are uniform; under a real effect they pile up near zero.
p_null = null_pvalues(1)[:, 0]
p_alt = stats.ttest_ind(rng.normal(0.5, 1, (reps, 30)), rng.normal(0, 1, (reps, 30)), axis=1).pvalue
bins = np.linspace(0, 1, 11)
print("\np-value bin null true effect of 0.5 sd (n = 30 per group)")
for lo, hi, c0, c1 in zip(bins[:-1], bins[1:], np.histogram(p_null, bins)[0], np.histogram(p_alt, bins)[0]):
print(f"{lo:.1f} to {hi:.1f} {c0:6} {'#' * (c0 // 100):10} {c1:6} {'#' * (c1 // 100)}")The simulated chance of at least one false positive matches , reaching 64% for twenty tests and 92% for fifty, and the average number of false "discoveries" is simply . The histograms show the uniform null distribution — about 1,000 in every bin of width 0.1 — and the pile-up near zero when there is a moderate effect. Even then, with 30 per group, only about 60% of p-values fall below 0.1: real effects do not guarantee small p-values when samples are small.
Error rates for families of tests
Suppose we test null hypotheses, of which are true. Let be the number we reject ("discoveries") and the number of those that are true nulls (false discoveries).
Definition— family-wise error rate and false discovery rate
The family-wise error rate is , the probability of making even one false discovery. The false discovery proportion is , the fraction of discoveries that are false, and the false discovery rate is its expectation, .
Controlling the FWER at 5% means that, across the whole family, there is at most a 5% chance of a single false claim. Controlling the FDR at 5% means that, on average, at most 5% of the claims made are false: a looser promise that permits more discoveries.
Theorem— Bonferroni correction
If each of hypotheses is rejected when its p-value is at most , then , whatever the dependence between the tests.
Proof
Let be the event that true null is rejected; since its p-value is uniform (or stochastically larger), . By the union bound, .
∎
Bonferroni is simple and valid under any dependence, but it is conservative, and with many tests it has little power. Two refinements are standard. Order the p-values .
- Holm's step-down procedure (1979): compare with , with , with , and so on; reject until the first p-value that fails its threshold, and stop there. It also controls the FWER under any dependence, and it rejects everything Bonferroni rejects and sometimes more, so there is no reason to prefer plain Bonferroni.
- The Benjamini–Hochberg (BH) procedure (1995) for FDR level : find the largest with , and reject the hypotheses with the smallest p-values. Yoav Benjamini and Yosef Hochberg proved that for independent tests ; the guarantee also holds under the positive dependence common in practice, and the more cautious Benjamini–Yekutieli variant, which replaces by , holds under any dependence.
Example— Three procedures, one list
The figure's 20 p-values begin , with .
- Bonferroni rejects : the first three.
- Holm rejects (each below , , ), then stops, because exceeds : three.
- BH compares with : , but and no later p-value comes back under its line, so : eight discoveries. BH rejects everything up to the largest rank that passes, even if some smaller rank had failed its own comparison; here none does.
The procedures compared
The program implements all three procedures and applies them to 500 simulated "screens", each testing 1,000 hypotheses of which 100 are real effects (a statistic centred at 3 instead of 0). For each method it records the number of discoveries, the number of false ones, the average false discovery proportion (the FDR), the FWER and the power (the fraction of real effects found). It then checks the BH implementation against SciPy's.
import numpy as np
from scipy import stats
def bonferroni(p, alpha=0.05):
return p <= alpha / len(p)
def holm(p, alpha=0.05):
m = len(p)
order = np.argsort(p)
reject = np.zeros(m, dtype=bool)
for i, j in enumerate(order): # i = 0, 1, ...: compare with alpha / (m - i)
if p[j] > alpha / (m - i):
break # stop at the first failure
reject[j] = True
return reject
def benjamini_hochberg(p, q=0.05):
m = len(p)
order = np.argsort(p)
below = p[order] <= q * np.arange(1, m + 1) / m
k = np.nonzero(below)[0].max() + 1 if below.any() else 0 # largest i with p_(i) <= i q / m
reject = np.zeros(m, dtype=bool)
reject[order[:k]] = True # reject the k smallest p-values
return reject
# 1,000 hypotheses, of which 100 are real effects. Each test is a z-test: z ~ N(effect, 1).
rng = np.random.default_rng(1995)
m, m1, reps = 1_000, 100, 500
is_real = np.arange(m) < m1
results = {name: [] for name in ["none (p <= 0.05)", "Bonferroni", "Holm", "Benjamini-Hochberg"]}
for _ in range(reps):
z = rng.normal(np.where(is_real, 3.0, 0.0), 1.0)
p = 2 * stats.norm.sf(np.abs(z))
for name, rej in [("none (p <= 0.05)", p <= 0.05), ("Bonferroni", bonferroni(p)),
("Holm", holm(p)), ("Benjamini-Hochberg", benjamini_hochberg(p))]:
V, R = np.sum(rej & ~is_real), np.sum(rej)
results[name].append((R, V, V / max(R, 1), V > 0, np.sum(rej & is_real) / m1))
print(f"{'method':20} {'discoveries':>11} {'false':>6} {'FDR':>6} {'FWER':>6} {'power':>6} (averages over {reps} replications)")
for name, rows in results.items():
R, V, fdp, any_false, power = np.array(rows, dtype=float).mean(axis=0)
print(f"{name:20} {R:11.1f} {V:6.1f} {fdp:6.3f} {any_false:6.2f} {power:6.2f}")
# Check our Benjamini-Hochberg against SciPy's adjusted p-values on one data set.
z = rng.normal(np.where(is_real, 3.0, 0.0), 1.0)
p = 2 * stats.norm.sf(np.abs(z))
same = np.array_equal(benjamini_hochberg(p), stats.false_discovery_control(p) <= 0.05)
print("\nagrees with scipy.stats.false_discovery_control:", same)Without correction the screen "finds" 130 effects, of which 45 are false — a third of the list. Bonferroni and Holm make a false discovery in only 3% of screens, but find just 15% of the real effects. Benjamini–Hochberg finds half the real effects, about 52 discoveries per screen, at the cost of about 2.5 false ones: an FDR of 0.047, matching the theory, , within simulation error. It almost always makes at least one false discovery (an FWER of 0.88), which is exactly the trade it was designed to make.
The garden of forking paths
Formal corrections assume we know how many tests were run. Often we do not, because the multiplicity is hidden in choices made while analysing the data. Andrew Gelman and Eric Loken call this the garden of forking paths: each reasonable decision — which outcome to emphasise, which participants to include, whether to adjust for a covariate, when to stop collecting data — is a fork, and a researcher who takes the forks after seeing the data can usually find a path to . No dishonesty is needed; every individual choice can be defensible. Joseph Simmons, Leif Nelson and Uri Simonsohn showed in 2011 how a few such researcher degrees of freedom could make almost any hypothesis "significant", and called the result false-positive psychology. The practice of trying analyses until one works is often called p-hacking.
The program simulates that garden when the treatment has no effect at all. Each study has 20 participants per group (extendable to 30), two correlated outcome measures, and participants from a morning and an afternoon session.
import itertools
import numpy as np
from scipy import stats
rng = np.random.default_rng(2011)
reps = 20_000
def ttest_p(x, y, mx, my):
"""Two-sided Student t-test p-values, row by row, using only the entries where the masks are True."""
nx, ny = mx.sum(1), my.sum(1)
xm, ym = (x * mx).sum(1) / nx, (y * my).sum(1) / ny
ssx = (((x - xm[:, None]) ** 2) * mx).sum(1)
ssy = (((y - ym[:, None]) ** 2) * my).sum(1)
sp2 = (ssx + ssy) / (nx + ny - 2)
t = (xm - ym) / np.sqrt(sp2 * (1 / nx + 1 / ny))
return 2 * stats.t.sf(np.abs(t), nx + ny - 2)
# No real effect anywhere. Two outcome measures per participant, correlated 0.5; up to 30 per group.
cov = [[1, 0.5], [0.5, 1]]
g1 = rng.multivariate_normal([0, 0], cov, size=(reps, 30))
g2 = rng.multivariate_normal([0, 0], cov, size=(reps, 30))
outcomes = {"outcome A": (g1[..., 0], g2[..., 0]), "outcome B": (g1[..., 1], g2[..., 1]),
"average": (g1.mean(-1), g2.mean(-1))}
session = rng.integers(0, 2, size=(2, reps, 30)).astype(bool) # morning or afternoon session
def p_for(outcome, subgroup, n):
x, y = outcomes[outcome][0][:, :n], outcomes[outcome][1][:, :n]
if subgroup == "all":
mx, my = np.ones_like(x, bool), np.ones_like(y, bool)
else: # analyse one session only
want = subgroup == "morning"
mx, my = session[0][:, :n] == want, session[1][:, :n] == want
return ttest_p(x, y, mx, my)
def false_positive_rate(outcome_set, subgroup_set, extend):
significant = np.zeros(reps, bool)
for o, s in itertools.product(outcome_set, subgroup_set):
significant |= p_for(o, s, 20) < 0.05
if extend: # "not quite significant? add 10 per group"
significant |= p_for(o, s, 30) < 0.05
return significant.mean()
all_o, all_s = list(outcomes), ["all", "morning", "afternoon"]
print("choices available to the analyst false-positive rate")
print(f" one analysis, fixed in advance {false_positive_rate(['outcome A'], ['all'], False):.3f}")
print(f" pick among outcome A, outcome B or their average {false_positive_rate(all_o, ['all'], False):.3f}")
print(f" everyone, or the morning or afternoon session only {false_positive_rate(['outcome A'], all_s, False):.3f}")
print(f" add 10 per group if not yet significant {false_positive_rate(['outcome A'], ['all'], True):.3f}")
print(f" all three kinds of choice (18 paths) {false_positive_rate(all_o, all_s, True):.3f}")
# Peeking: test after every 10 participants per group, up to 100, and stop as soon as p < 0.05.
x, y = rng.normal(size=(reps, 100)), rng.normal(size=(reps, 100))
stopped = np.zeros(reps, bool)
for n in range(10, 101, 10):
stopped |= stats.ttest_ind(x[:, :n], y[:, :n], axis=1).pvalue < 0.05
print(f"\npeeking 10 times, stopping at the first p < 0.05: {stopped.mean():.3f}")Each single analysis is honest: a fixed, pre-planned test gives false positives 4.9% of the time. Allowing a choice among three outcome measures nearly doubles the rate, to 9.4%; analysing either everyone or one session gives 11.4%; adding ten participants per group when the first result is "nearly significant" gives 7.8%. Together, the 18 paths give 31.6% — more than six times the nominal rate — with no fabricated data and no single indefensible choice. Optional stopping, which you met in Hypothesis testing, is powerful on its own: testing after every 10 participants per group, up to 100, and stopping at the first , produces a false positive almost 20% of the time.
There are honest ways to look at data repeatedly. Group sequential designs, standard in clinical trials, plan a fixed number of interim analyses and spend the error rate across them (for example, with very strict early thresholds, as in the O'Brien–Fleming design). Newer anytime-valid methods, based on e-values and test martingales, remain valid however often you look and whenever you stop, and some online experimentation platforms now use them.
The replication crisis
In 2005 John Ioannidis published an essay with a deliberately provocative title, "Why most published research findings are false". Its argument is arithmetic, and the last program makes it concrete. Suppose a field tests many hypotheses of which only 10% are true, with true effects of 0.4 standard deviations — a typical size in the social and biomedical sciences — and journals publish the studies with . Each simulated study is also replicated once, exactly, with the same design.
import numpy as np
from scipy import stats
rng = np.random.default_rng(2015)
def literature(n_per_group, share_true=0.10, effect=0.4, studies=100_000):
"""Simulate many two-group studies; 'publish' those with p < 0.05; replicate them once, same design."""
true = rng.uniform(size=studies) < share_true
d = np.where(true, effect, 0.0) # true standardised effect
se = np.sqrt(2 / n_per_group) # standard error of the estimated effect
est = rng.normal(d, se)
published = 2 * stats.norm.sf(np.abs(est / se)) < 0.05
rep = rng.normal(d, se) # an independent replication of each study
replicated = (2 * stats.norm.sf(np.abs(rep / se)) < 0.05) & (np.sign(rep) == np.sign(est))
pub_true = published & true
return {
"power": np.mean(published[true]),
"share of published findings that are true": np.mean(true[published]),
"median published estimate (true effects only)": np.median(np.abs(est[pub_true])),
"replication rate of published findings": np.mean(replicated[published]),
}
print("10% of tested hypotheses are true, each with a true effect of 0.40 standard deviations")
for n in [20, 50, 100]:
print(f"\n{n} participants per group:")
for k, v in literature(n).items():
print(f" {k:46} {v:.2f}")With 20 participants per group, the studies have power 0.25, and only 36% of the published findings are true. The true effects that get published are exaggerated: their median estimate is 0.77, nearly double the real 0.40, because with low power only the studies that overestimated the effect by luck crossed the threshold — the winner's curse, or the type M error of Hypothesis testing. And only 10% of published findings replicate. With 100 per group (power 0.81), 64% of published findings are true, the exaggeration nearly disappears, and the replication rate rises to 53%. None of this requires fraud or even forking paths; low power and a filter on significance are enough. Adding forking paths makes every number worse.
Large replication projects have since measured what happens in practice:
- The Reproducibility Project: Psychology (Open Science Collaboration, 2015) repeated 100 studies from three psychology journals: 97% of the originals had reported significant results, against 36% of the replications, and the replication effect sizes were on average about half the originals'.
- Colin Camerer, Anna Dreber and colleagues replicated 18 laboratory experiments in economics (2016), of which 11 (61%) found a significant effect in the original direction, and 21 social-science experiments published in Nature and Science (2018), of which 13 (62%) did.
- The Reproducibility Project: Cancer Biology (Errington and colleagues, 2021) could complete only 50 of the experiments it had planned, from 23 papers — often because the original methods were not described in enough detail to repeat. For positive effects, the median replication effect size was 85% smaller than the original.
- The Brazilian Reproducibility Initiative repeated experiments from Brazilian biomedical papers that used three common laboratory methods, in dozens of laboratories across the country, and reported in a 2025 preprint that fewer than half replicated by any of the criteria it used.
These results need careful reading. A failed replication does not prove that the original finding was false: the replication may have had a different population or a subtly different procedure, or the effect may genuinely vary between settings, and critics of some projects argued exactly that. Conversely, one successful replication does not prove an effect. Replication rates also differ between fields, and fields that adopted strict thresholds and routine replication earlier, such as human genetics after the move to genome-wide studies, tend to fare better. But the overall pattern — inflated effects and lower-than-expected replication rates in fields that relied on small studies and the 0.05 threshold — is now widely accepted, including by many of the researchers whose work was examined. That acceptance, and the reforms that followed, are science correcting itself.
Good practice
The reforms of the last fifteen years attack the problem at its sources.
- Preregistration. Before collecting or analysing data, write down the hypotheses, the primary outcome, the sample size and stopping rule, and the analysis plan, and file them with a time stamp on a public registry such as the Open Science Framework or AsPredicted. This closes the forking paths in advance. It does not forbid exploration: analyses not in the plan are reported as exploratory, to be confirmed by new data. Clinical trials have had to be registered before enrolment, on ClinicalTrials.gov or a registry in the WHO network, as a condition of publication in major medical journals since 2005, and outcome switching in trials is now tracked and criticised.
- Registered reports. A journal reviews the introduction and methods before the results exist, and commits to publish the study whatever it finds, if it is carried out as planned. This removes publication bias at its root. Anne Scheel, Mitchell Schijen and Daniël Lakens found that 44% of registered reports in psychology supported their first hypothesis, against 96% of standard articles — evidence of how strongly the usual literature had been filtered.
- Power and estimation. Plan sample sizes for adequate power for effects that matter, report effect sizes with confidence or credible intervals rather than bare p-values, and treat a single study as one piece of evidence to be combined, by meta-analysis, with others.
- Transparency. Share data (where privacy and consent allow), code and materials, so that others can reproduce the analysis and repeat the study. Report every outcome and every analysis tried. A multiverse analysis runs all the reasonable paths and shows how the conclusion varies across them.
- Blind analysis. In particle physics and cosmology, analysts often finalise their methods with the key result hidden, and unblind only once — a discipline that removes the temptation to steer.
- Modelling many effects together. A Bayesian multilevel model estimates many related effects jointly and shrinks noisy estimates towards each other, which tempers the winner's curse without a separate correction; the Advanced level's Multilevel models lesson shows how.
The role of p-values themselves is still debated, and fairly so. The American Statistical Association's 2016 statement set out six principles, among them that a p-value does not measure the probability that a hypothesis is true, nor the size or importance of an effect, and that scientific conclusions should not rest on whether a p-value crosses a threshold. Some researchers have proposed lowering the default threshold for new discoveries to 0.005; others argue that each study should justify its own threshold from the costs of errors; others still that fixed thresholds should be abandoned in favour of reporting estimates and their uncertainty. All three camps agree that a lone , from an analysis chosen after seeing the data, is weak evidence.
Optional Christian reflection · Just weights
A reflection from the Christian tradition that inspires Lumenary Academy. It is not part of the mathematics and you are welcome to skip it.
"A false balance is abomination to the Lord: but a just weight is his delight" (Proverbs 11:1, King James Version). A scale that reads a little heavy cheats every customer a little, and nobody sees it. An analysis steered towards significance is a false balance of the same kind, and preregistration, sharing data and reporting every result are ways of letting others check our scales. Honest reporting of a null result can feel like failure, but it is a just weight, and the people who will rely on the science deserve one.
Common misconceptions
Misconception
"Corrections are needed only when a paper reports many tests." Multiplicity also comes from analyses that were tried and not reported, and from choices that would have been made differently with different data. Preregistration addresses what corrections cannot see.
Misconception
"Bonferroni is always the safe choice." It controls the FWER, but in a large screen it sacrifices most of the power, as the simulation showed (15% against 50% for BH). Holm is uniformly better than Bonferroni for FWER control, and FDR control is often the more useful goal for exploratory screens.
Misconception
"With FDR controlled at 5%, each discovery has a 95% chance of being real." The 5% is an average over the whole list. The discoveries with the smallest p-values are much more trustworthy than those near the cut-off, which may well be false more often than 5% of the time.
Misconception
"A failed replication proves the original finding was false." It is evidence, weighed together with the original. Both studies are noisy; differences in population and method matter; and a well-powered replication with a precise estimate near zero is more informative than a small one that merely failed to reach significance.
Misconception
"Preregistration means you cannot explore the data." Exploration is welcome and often where discoveries begin. Preregistration only asks that confirmatory and exploratory analyses be labelled honestly, so readers know which p-values carry their nominal meaning.
Exercises
1. Multiple testing and reproducibility
Warm-upHumanA study runs 10 independent tests at level 0.05, and every null hypothesis is true. What is the probability of at least one false positive, as a percentage rounded to the nearest whole number?
2. Multiple testing and reproducibility
CoreHumanTen hypotheses gave these p-values:
| Hypothesis | H1 | H2 | H3 | H4 | H5 | H6 | H7 | H8 | H9 | H10 |
|---|---|---|---|---|---|---|---|---|---|---|
| p-value | 0.041 | 0.001 | 0.21 | 0.011 | 0.006 | 0.77 | 0.024 | 0.004 | 0.48 | 0.029 |
Apply the Benjamini–Hochberg procedure with . Which hypotheses are rejected? Enter their labels (for example H2).
3. Multiple testing and reproducibility
CoreHumanFor the same ten p-values,
| Hypothesis | H1 | H2 | H3 | H4 | H5 | H6 | H7 | H8 | H9 | H10 |
|---|---|---|---|---|---|---|---|---|---|---|
| p-value | 0.041 | 0.001 | 0.21 | 0.011 | 0.006 | 0.77 | 0.024 | 0.004 | 0.48 | 0.029 |
how many hypotheses does Holm's procedure reject at ?
4. Multiple testing and reproducibility
CoreHumanA laboratory tests 1,000 hypotheses. 100 of them are true, and its studies have power 0.8 to detect them; the other 900 are false, tested at . Only results with are published. What percentage of the published findings are true? Round to the nearest whole number.
5. Multiple testing and reproducibility
CoreHumanWhich of these is an example of the garden of forking paths?
6. Multiple testing and reproducibility
CoreHumanWhich statements about preregistration and registered reports are correct? Select all that apply.
7. Multiple testing and reproducibility
CoreComputationalWrite benjamini_hochberg(pvals, q=0.05) that returns a boolean NumPy array, in the original order of pvals, that is True for the hypotheses the Benjamini–Hochberg procedure rejects at FDR level q: sort the p-values, find the largest rank with , and reject the smallest. The tests include the ten p-values from the earlier exercise, ties, and 200 random cases checked against scipy.stats.false_discovery_control.
8. Multiple testing and reproducibility
StretchComputationalWrite holm(pvals, alpha=0.05) that returns a boolean NumPy array, in the original order, that is True for the hypotheses Holm's step-down procedure rejects: visit the p-values from smallest to largest, compare the -th smallest () with , reject while the comparison succeeds, and stop at the first failure. The tests check hand-worked cases and 300 random cases against Holm-adjusted p-values.
Further reading and resources
- Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing · Yoav Benjamini and Yosef Hochberg · 1995
- A Simple Sequentially Rejective Multiple Test Procedure · Sture Holm · 1979
- False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant · Joseph Simmons, Leif Nelson and Uri Simonsohn · 2011
- The Statistical Crisis in Science · Andrew Gelman and Eric Loken · 2014The garden of forking paths, for a general audience.
- Why Most Published Research Findings Are False · John Ioannidis · 2005
- Estimating the reproducibility of psychological science · Open Science Collaboration · 2015
- Investigating the replicability of preclinical cancer biology · Timothy Errington and others · 2021
- The ASA Statement on p-Values: Context, Process, and Purpose · Ronald Wasserstein and Nicole Lazar · 2016
- The preregistration revolution · Brian Nosek, Charles Ebersole, Alexander DeHaven and David Mellor · 2018
- Registered Reports · Center for Open ScienceHow the format works, and a list of journals that offer it.