Skip to content
LumenaryAcademy
Frontier 80 min

The statistics of evaluating AI

Benchmarks are samples and scores are estimates — confidence intervals with clustered questions, pass rates done right, fair paired comparisons, the winner's curse and contamination, human and model judges, and how to combine them.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

A leaderboard is a table of estimates

A new model scores 71.3% on a reasoning benchmark; the previous best scored 70.1%. A press release calls it the new state of the art, a team switches its product to it, and a policy brief cites the gain as evidence of rapid progress. Every one of those decisions treats 71.3 and 70.1 as if they were measured exactly.

They were not. The benchmark is a few hundred or a few thousand questions chosen from an endless space of questions anyone cares about; the model's answers are sampled at a temperature; the grading is done by a script, a person or another model, each with its own errors. A score is an estimate, and everything you learned in Estimation and confidence intervals, Comparing groups and Multiple testing and reproducibility applies to it — with some twists that make AI evaluation statistically interesting in its own right.

Lumenary CS's Evaluating models lesson introduced the benchmark landscape, standard errors, the paired bootstrap and the biases of human and model judges. This lesson is the statistician's companion to it: where the variance in a score comes from, how to get intervals that are honest when questions are clustered and answers are sampled, how to compare models with the power the data allow, how benchmarks are overfitted, and how to combine scarce human judgements with plentiful machine ones. Lumenary Robotics' Evaluation, safety and deployment applies the same ideas to robots, where every trial is expensive.

IntuitionH

Where the variance comes from

A benchmark score passes through several random stages, and each adds variation:

the tasks you actually care aboutbenchmark: a sample of passages or tasksquestions within each passagethe model's sampled answers (temperature, seed)the grader: a person, a test suite, or a model judgeis the benchmark representative? (validity)sampling of clusters: dominant, often ignoredquestions in a cluster are correlatedwithin-question randomness: average k samplesgrader error and bias: calibrate on human labels
The stages a benchmark score passes through. Each arrow adds variation or bias. The usual binomial standard error accounts only for questions treated as independent; clustering, repeated sampling and grader error each need their own treatment.

Three distinctions organise the rest of the lesson.

  • Which population? If you care only about this exact benchmark, the questions are fixed and the only randomness is in the model's answers and the grading. If you care about the kind of question the benchmark represents — the usual case — the questions are a sample and their sampling variance dominates.
  • Independent or clustered? Many benchmarks group questions: several questions about one reading passage, several test cases for one coding problem, several turns of one conversation, translations of one item into many languages. Questions in a group share difficulty, so they are correlated, and treating them as independent understates the uncertainty.
  • Fixed or sampled answers? At temperature above zero, the model's answer to a question is itself random. Sampling kk answers per question and averaging reduces this within-question variance — but no amount of resampling reduces the between-question variance.
ComputationC

Confidence intervals with clustered questions

Definition— Cluster-robust standard error of an accuracy

Let a benchmark have CC clusters (passages, problems, conversations), with xcj∈{0,1}x_{cj} \in \{0, 1\} the correctness of the answer to question jj in cluster cc. If clusters are sampled independently, the standard error of the overall accuracy p^\hat p should be computed from the cluster-level totals, not from the individual answers. With equal cluster sizes it is simply the standard deviation of the cluster means divided by C\sqrt C. The ratio of the clustered to the naive variance is the design effect; with clusters of size mm and intra-cluster correlation ρ\rho, it is approximately 1+(m−1)ρ1 + (m - 1)\rho.

This is the same correction that survey statisticians make for cluster samples (Samples and surveys) and that Multilevel models makes for pupils within schools. The simulation below builds a reading benchmark of 120 passages with 5 questions each, where passages differ in difficulty, and checks how often each kind of interval covers the accuracy over the whole population of passages.

python
import numpy as np

rng = np.random.default_rng(3)
n_passages, per_passage = 120, 5                 # a reading benchmark: 5 questions about each of 120 passages
a, s = 0.9, 1.3                                  # model skill; spread of passage difficulty (logit scale)

def run_benchmark():
    """One draw of the benchmark: new passages, new questions, the model answers each once."""
    u = rng.normal(0, s, n_passages)                                   # passage difficulty, shared by its questions
    p = 1 / (1 + np.exp(-(a - u[:, None] + rng.normal(0, 0.5, (n_passages, per_passage)))))
    return (rng.random(p.shape) < p).astype(float)                     # 1 = correct; shape (passages, questions)

# The quantity we want to know: accuracy over the whole population of passages and questions.
z = rng.normal(size=2_000_000)
true_acc = np.mean(1 / (1 + np.exp(-(a + np.sqrt(s**2 + 0.5**2) * z))))

def intervals(x):
    n = x.size
    acc = x.mean()
    se_naive = np.sqrt(acc * (1 - acc) / n)                            # pretends all 600 answers are independent
    cluster_means = x.mean(axis=1)
    se_cluster = cluster_means.std(ddof=1) / np.sqrt(x.shape[0])       # treats passages as the sampling units
    return acc, se_naive, se_cluster

x = run_benchmark()
acc, se_n, se_c = intervals(x)
print(f"one benchmark run: accuracy {acc:.3f} on {x.size} questions; population accuracy {true_acc:.3f}")
print(f"  naive 95% CI     [{acc - 1.96 * se_n:.3f}, {acc + 1.96 * se_n:.3f}]   SE {se_n:.4f}")
print(f"  clustered 95% CI [{acc - 1.96 * se_c:.3f}, {acc + 1.96 * se_c:.3f}]   SE {se_c:.4f}")
print(f"  design effect (variance ratio) {(se_c / se_n) ** 2:.2f}: the 600 answers carry the information "
      f"of about {x.size * (se_n / se_c) ** 2:.0f} independent ones")

cover = np.array([[abs(r[0] - true_acc) <= 1.96 * r[1], abs(r[0] - true_acc) <= 1.96 * r[2]]
                  for r in (intervals(run_benchmark()) for _ in range(4000))])
print(f"\nover 4,000 simulated benchmarks, the nominal 95% interval covers the population accuracy:")
print(f"  naive {cover[:, 0].mean():.3f}    clustered {cover[:, 1].mean():.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.
0.600.620.640.660.680.700.720.74accuracypopulation accuracy 0.659naive: SE 0.019clustered: SE 0.028
The two 95% intervals from the run above. Both contain the population accuracy here, but over many simulated benchmarks the naive interval misses it 14% of the time instead of 5%.

Two practical notes. The cluster-robust formula is itself an estimate and needs a reasonable number of clusters — with fewer than about 30, use small-sample corrections or a cluster bootstrap that resamples whole passages. And for small benchmarks of either kind, intervals based on the normal approximation are unreliable near 0% or 100% accuracy; Bowyer, Aitchison and Ivanova argue for Bayesian or exact intervals (such as Wilson's or Clopper–Pearson) when there are fewer than a few hundred questions. Miller (2024) and Madaan et al. (2024) give practical recipes and measurements of how large these effects are on real benchmarks.

ComputationC

Pass rates: pass@k and pass^k

For code generation and many agent tasks, a model is allowed several attempts and a test suite decides each one. pass@k is the probability that at least one of kk attempts succeeds. The obvious way to estimate it — generate exactly kk samples per task and record whether any passed — is wasteful and noisy. Chen et al. (2021) generate n≥kn \ge k samples per task, count the cc that pass, and use

pass@k^=1−(n−ck)(nk),\widehat{\text{pass@}k} = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}},

the probability that kk samples drawn without replacement from the nn include at least one success. It is unbiased for each task's pass@k, and its average over tasks is unbiased for the benchmark's. The tempting plug-in 1−(1−c/n)k1 - (1 - c/n)^k is biased downwards, because (1−p)k(1 - p)^k is a convex function of pp (Jensen's inequality). For agents that must succeed every time — a customer-service agent that handles the same request for many users — the opposite metric matters: pass^k, the probability that all kk attempts succeed, estimated by (ck)/(nk)\binom{c}{k} / \binom{n}{k} (Yao et al., 2024).

python
import numpy as np
from math import comb

rng = np.random.default_rng(8)
n_tasks, n_samples = 150, 20                         # 150 coding tasks, 20 sampled solutions per task
p_task = rng.beta(0.6, 0.9, n_tasks)                 # each task's true per-sample success rate (unknown)

def pass_at_k(n, c, k):
    """Unbiased estimator (Chen et al., 2021): chance that at least one of k samples, drawn
    without replacement from the n we generated, is correct."""
    return 1.0 if n - c < k else 1 - comb(n - c, k) / comb(n, k)

def pass_hat_k(n, c, k):
    """Chance that all k of k samples are correct: a measure of reliability, not of luck."""
    return comb(c, k) / comb(n, k)

for k in [1, 5, 10]:
    truth = np.mean(1 - (1 - p_task) ** k)
    unbiased, naive = [], []
    for _ in range(400):                              # repeat the sampling of solutions many times
        c = rng.binomial(n_samples, p_task)           # correct solutions per task
        unbiased.append(np.mean([pass_at_k(n_samples, ci, k) for ci in c]))
        naive.append(np.mean(1 - (1 - c / n_samples) ** k))   # plug in c/n: biased low by Jensen
    print(f"pass@{k:<2}  true {truth:.4f}   unbiased estimator {np.mean(unbiased):.4f}   plug-in {np.mean(naive):.4f}")

# One evaluation run, with a bootstrap over tasks (the tasks are the sample).
c = rng.binomial(n_samples, p_task)
print()
for k in [1, 10]:
    per_task = np.array([pass_at_k(n_samples, ci, k) for ci in c])
    boot = [per_task[rng.integers(0, n_tasks, n_tasks)].mean() for _ in range(4000)]
    lo, hi = np.quantile(boot, [0.025, 0.975])
    print(f"one run: pass@{k} = {per_task.mean():.3f}, 95% bootstrap CI over tasks [{lo:.3f}, {hi:.3f}]")
print(f"pass^5 (all five samples correct) = {np.mean([pass_hat_k(n_samples, ci, 5) for ci in c]):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The unbiased estimator matches the truth (0.8217 against 0.8209 for pass@10) while the plug-in falls short by almost three points. The bootstrap resamples tasks, the units sampled from the population; the 20 solutions per task reduce within-task noise but are not 20 independent observations of the benchmark. And the last line shows why both metrics are needed: the same model passes at least one of five attempts on most tasks, but passes all five on only 12.5% of them.

ComputationC

Comparing two models fairly

When two models answer the same questions, their results are paired, and the analysis should use the pairing. Only questions on which the models disagree carry information about which is better; questions both get right, or both get wrong, cancel out. McNemar's test makes this exact: under the null hypothesis that the models are equally good, each discordant question is equally likely to favour either model, so the number favouring model A is binomial with probability 12\tfrac12. For scores that are averages over several samples or graders, the paired tt-test on per-question differences plays the same role.

python
import numpy as np
from scipy import stats

rng = np.random.default_rng(21)
n = 500

def two_models():
    """Two models answer the same n questions. Difficulty is shared: hard questions are hard for both."""
    difficulty = rng.normal(0, 3.0, n)
    a = (rng.random(n) < 1 / (1 + np.exp(-(1.0 - difficulty)))).astype(int)   # model A, 1 = correct
    b = (rng.random(n) < 1 / (1 + np.exp(-(0.6 - difficulty)))).astype(int)   # model B, a little weaker
    return a, b

def tests(a, b):
    diff = a.mean() - b.mean()
    se_unpaired = np.sqrt(a.var() / n + b.var() / n)          # as if two separate question sets
    se_paired = (a - b).std(ddof=1) / np.sqrt(n)              # SE of the mean per-question difference
    n10, n01 = int(np.sum(a > b)), int(np.sum(a < b))         # discordant questions
    p_mcnemar = stats.binomtest(n10, n10 + n01, 0.5).pvalue   # exact: discordant pairs are fair coin flips under H0
    return diff, se_unpaired, se_paired, n10, n01, p_mcnemar

a, b = two_models()
diff, se_u, se_p, n10, n01, p_mc = tests(a, b)
print(f"accuracy A {a.mean():.3f}, B {b.mean():.3f}, difference {diff:+.3f}")
print(f"discordant questions: A right & B wrong {n10}, A wrong & B right {n01} (of {n})")
print(f"SE of the difference: unpaired {se_u:.4f}, paired {se_p:.4f}")
print(f"p-values: unpaired z-test {2 * stats.norm.sf(diff / se_u):.3f}, paired z-test {2 * stats.norm.sf(diff / se_p):.3f}, "
      f"McNemar exact {p_mc:.3f}")

# Power: how often does each test detect the (real) difference at the 5% level?
reject = np.array([[2 * stats.norm.sf(abs(d / su)) < 0.05, 2 * stats.norm.sf(abs(d / sp)) < 0.05, pm < 0.05]
                   for d, su, sp, _, _, pm in (tests(*two_models()) for _ in range(2000))])
print(f"power over 2,000 replications: unpaired {reject[:, 0].mean():.2f}, paired {reject[:, 1].mean():.2f}, "
      f"McNemar {reject[:, 2].mean():.2f}")

# Planning: questions needed to detect a 3-point gain with 80% power when 20% of questions are discordant.
delta, disc = 0.03, 0.20
print(f"\nquestions needed for +{delta:.2f} at 80% power, discordance {disc:.0%}: about "
      f"{(stats.norm.ppf(0.975) + stats.norm.ppf(0.80)) ** 2 * disc / delta**2:.0f}")

# The winner's curse: 20 prompt variants, all truly 70% accurate, compared on one fixed test set.
best_seen, best_fresh = [], []
for _ in range(2000):
    scores = rng.binomial(n, 0.70, 20) / n
    best_seen.append(scores.max())                        # the variant we would report
    best_fresh.append(rng.binomial(n, 0.70) / n)          # the same variant on fresh questions
print(f"best of 20 equal variants: reported {np.mean(best_seen):.3f}, on fresh questions {np.mean(best_fresh):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Model A is truly better, by 4.5 points over the population of questions (computed separately); in this run it scores 3.6 points higher. The paired standard error (0.0217) is about a third smaller than the unpaired one (0.0313). Over 2,000 replications the unpaired test detects the difference only 23% of the time; the paired tests about 55%. Pairing more than doubles the power, at no cost. Even so, 500 questions are not enough to detect a real 4.5-point gain reliably: the planning formula n≈(z0.975+z0.8)2 pdisc/δ2n \approx (z_{0.975} + z_{0.8})^2\, p_{\text{disc}} / \delta^2 says a 3-point gain with 20% of questions discordant needs about 1,744 questions for 80% power.

The last lines of output show a subtler problem. Twenty prompt variants, all truly 70% accurate, are tried on the same 500-question test set, and the best is reported: 73.7%. On fresh questions it scores 70.1%. This is the winner's curse, the selection effect of Multiple testing and reproducibility in evaluation clothing. It afflicts prompt engineering, hyperparameter search, checkpoint selection and leaderboard submissions alike, and its only cure is a final evaluation on data that played no part in any choice.

ExplanationH

Overfitting to benchmarks, and contamination

Two different mechanisms make benchmark scores optimistic, and they need different statistical thinking.

Adaptive overfitting. A community that repeatedly evaluates on the same public test set, and publishes what works, is running the winner's curse at scale. Theory says this can be serious: Dwork et al. (2015) showed that the number of adaptively chosen queries a holdout set can answer reliably is limited, and that adding noise to the answers — a technique borrowed from differential privacy, the subject of the next lesson — extends it; Blum and Hardt (2015) designed a leaderboard ("the Ladder") that only reports significant improvements. The evidence is more reassuring than the theory. When Recht, Roelofs, Schmidt and Shankar (2019) built a fresh ImageNet test set following the original procedure, accuracies fell by 11 to 14 percentage points — but the ranking of models was almost perfectly preserved, and they attributed the drop mainly to a subtle shift in the new data's distribution rather than to adaptive overfitting. A decade of reuse had inflated absolute numbers less than feared, and comparisons hardly at all.

Contamination is a different failure: the test items, or close paraphrases, were in the training data, so the score partly measures memory. For language models trained on large web crawls it is common and hard to rule out. Statistically, contamination detection is an exercise in causal comparison:

  • Fresh replicate sets. Zhang et al. (2024) wrote a new set of grade-school maths problems matched in difficulty to a widely used public one. Some model families scored markedly lower on the new set, consistent with overfitting to the public one; the strongest models showed little gap. The comparison is only as good as the matching of difficulty, which itself has to be checked — for example by human solve rates.
  • Time splits. Benchmarks such as LiveCodeBench collect problems published after a model's training cutoff. A drop in performance on post-cutoff problems, relative to pre-cutoff ones, is evidence of contamination — but problems also change over time, so a careful analysis compares the drop with that of models whose cutoffs differ: a difference-in-differences design.
  • Held-out and private sets, kept by evaluators and never published, remain the most reliable protection.

Beyond both lies the question of validity: whether the benchmark measures the capability it is named after at all. Raji et al. (2021) argued that many benchmarks are presented as measures of general abilities they cannot support. Psychometricians' tools are arriving in AI evaluation: item response theory, which models each question's difficulty and discrimination, lets tinyBenchmarks estimate full-benchmark scores from about a hundred well-chosen items, and gives a principled way to see which questions actually separate models.

ComputationC

Human preferences, model judges, and combining them

For open-ended tasks there is no answer key, and evaluation relies on judgement — by people or by models. The CS lesson described the Bradley–Terry ratings and their biases in public arenas. Seen as a statistician, a preference evaluation has at least three random factors — prompts, raters and model samples — and its weaknesses are mostly familiar:

  • Who rates and what they are shown. Arena-style evaluations sample self-selected visitors and their own prompts; the population of prompts and raters defines what the rating means. Voters are known to favour longer, more formatted and more confident answers.
  • Rater disagreement. Two careful people often disagree about which of two good answers is better. Agreement should be measured (Cohen's κ\kappa or Krippendorff's α\alpha) and reported, and a rating difference smaller than the disagreement noise means little.
  • Uncertainty in ratings. Bradley–Terry strengths are estimates; their confidence intervals (by bootstrap over battles, or sandwich standard errors) are often wide enough that neighbouring models on a leaderboard are statistically tied. Rank intervals are more honest than ranks.
  • Selection and access. The Leaderboard Illusion (2025) argued that undisclosed private testing of many model variants, with only the best published, and unequal sampling rates across providers, can distort public rankings — the winner's curse again; the arena's operators disputed parts of the analysis.

Model judges — a strong model grading answers against a rubric — are cheap and fast, and agree with human experts about as often as experts agree with each other on some tasks (Zheng et al., 2023). They are also biased: by position, by length, towards answers that resemble their own (Wang et al., 2023). Averaging more judge labels does not remove a bias; it only makes the biased answer more precise.

Prediction-powered inference (Angelopoulos, Bates, Fannjiang, Jordan and Zrnic, 2023) is the principled way to have both: label everything with the judge, label a random subset with people as well, and use the subset to estimate and remove the judge's bias.

Definition— Prediction-powered estimate of a mean

With judge labels fif_i on all NN items and human labels yiy_i on a random sample of nn of them, the prediction-powered estimate of the mean human label is

θ^PP=1N∑i=1Nfi  +  1n∑i∈labelled(yi−fi),\hat\theta_{\text{PP}} = \frac1N \sum_{i=1}^N f_i \;+\; \frac1n \sum_{i \in \text{labelled}} (y_i - f_i),

the judge's average plus a rectifier that corrects its bias. It is unbiased whatever the judge's quality, and its variance is about Var⁡(f)/N+Var⁡(y−f)/n\operatorname{Var}(f)/N + \operatorname{Var}(y - f)/n: small when the judge usually agrees with people.

python
import numpy as np

rng = np.random.default_rng(13)
N, n = 10000, 300                 # responses graded by a model judge; of these, a random 300 also graded by people

def simulate():
    human = (rng.random(N) < 0.62).astype(float)           # the truth we care about: 62% of responses acceptable
    # The judge is cheap but imperfect: it rejects 5% of good answers and passes 15% of bad ones.
    judge = np.where(human == 1, rng.random(N) < 0.95, rng.random(N) < 0.15).astype(float)
    labelled = rng.choice(N, n, replace=False)              # a simple random sample gets human labels
    return human, judge, labelled

def estimates(human, judge, labelled):
    h, j = human[labelled], judge[labelled]
    out = {"human labels only": (h.mean(), h.std(ddof=1) / np.sqrt(n)),
           "judge only": (judge.mean(), judge.std(ddof=1) / np.sqrt(N))}
    # Prediction-powered inference: the judge's mean, corrected by the human-judge gap on the labelled sample.
    rectifier = h - j
    out["prediction-powered"] = (judge.mean() + rectifier.mean(),
                                 np.sqrt(judge.var(ddof=1) / N + rectifier.var(ddof=1) / n))
    return out

human, judge, labelled = simulate()
print(f"true acceptable rate among the {N} responses: {human.mean():.3f}")
for name, (est, se) in estimates(human, judge, labelled).items():
    print(f"{name:20s} {est:.3f}   95% CI [{est - 1.96 * se:.3f}, {est + 1.96 * se:.3f}]   width {2 * 1.96 * se:.3f}")

cover = {}
for _ in range(2000):
    human, judge, labelled = simulate()
    truth = human.mean()                                    # the acceptable rate among these N responses
    for name, (est, se) in estimates(human, judge, labelled).items():
        cover.setdefault(name, []).append(abs(est - truth) <= 1.96 * se)
print("\ncoverage of the nominal 95% interval over 2,000 repetitions:")
for name, c in cover.items():
    print(f"  {name:20s} {np.mean(c):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The judge alone gives a narrow interval in the wrong place: over 2,000 repetitions it never covers the true acceptable rate, because the judge passes 15% of bad answers. Human labels alone are unbiased but wide (width 0.109 from 300 labels). The prediction-powered interval is unbiased, covers 95.5% of the time, and is 40% narrower (width 0.065) — as precise as about (0.109/0.065)2≈2.8(0.109/0.065)^2 \approx 2.8 times as many human labels. PPI++ adds a tuning weight on the judge so that a useless judge can never make things worse. The same idea, a small gold-standard sample correcting a large cheap one, is a classic of survey sampling (the difference and regression estimators); its arrival in AI evaluation is recent. Its requirement is just as classic: the human-labelled subset must be a random sample, not the cases someone found interesting.

ExplanationH

Reporting an evaluation (2026)

As of late 2026, statistical practice in AI evaluation is improving but uneven. Some labs, government AI safety and security institutes, and academic evaluators report confidence intervals, paired comparisons and the number of samples; many papers and announcements still report bare point estimates to one decimal place. Open-source evaluation harnesses increasingly compute standard errors by default. A short checklist for reading or writing an evaluation:

  1. State the population. What tasks is the score meant to generalise to, and how were the benchmark's items chosen?
  2. Report uncertainty correctly. Intervals with the right unit of sampling (clusters, tasks), the number of items and the number of samples per item.
  3. Pair comparisons. Same items, same samples, paired tests, and the number of discordant items.
  4. Plan the size. Compute the questions needed for the smallest difference you care about, before running.
  5. Separate selection from reporting. Choose prompts, checkpoints and settings on a development set; report on a held-out set once.
  6. Address contamination with fresh, private or post-cutoff items, and say what was done.
  7. Calibrate the judge. If a model grades, measure its agreement with human labels on a random sample, and correct its bias (prediction-powered inference) rather than ignoring it.
  8. Say what the number does not measure. No benchmark measures everything its name suggests.
Common misconceptionsH

Common misconceptions

Misconception

"1,000 questions give a standard error of p(1−p)/1000\sqrt{p(1-p)/1000}." Only if they are independent. Questions sharing a passage, a problem or a template are correlated, and the effective sample size can be half the nominal one or less. Use cluster-robust intervals.

Misconception

"Sampling more answers per question makes the benchmark more precise without limit." Repeated samples reduce only the within-question variance. The between-question variance — how the benchmark's items happen to have been chosen — shrinks only with more questions.

Misconception

"The model with the higher score is better." Not unless the difference is large relative to its paired standard error, and the score was not the winner of a selection process on the same test set. Ties are common at the top of leaderboards.

Misconception

"A model judge with 90% agreement gives estimates accurate to within 10%." Agreement is not bias. A judge can agree 90% of the time and still shift the estimated rate systematically, in a direction that depends on which errors it makes. Measure its bias on a random human-labelled sample and correct for it.

Exercises

Exercises

1. The statistics of evaluating AI

Warm-upHuman

A benchmark has 1,000 questions in 100 groups of 10 (ten questions per source document). The intra-cluster correlation of correctness is ρ=0.2\rho = 0.2. Using the design effect 1+(m−1)ρ1 + (m - 1)\rho, how many independent questions is the benchmark worth? Round to the nearest whole number.

2. The statistics of evaluating AI

CoreHuman

For one task, a model generated n=10n = 10 solutions, of which c=3c = 3 pass the tests. Using the unbiased estimator 1−(n−ck)/(nk)1 - \binom{n-c}{k}/\binom{n}{k}, estimate pass@5 for this task, as a percentage rounded to the nearest whole number.

3. The statistics of evaluating AI

CoreHuman

You want to detect a 2-point accuracy gain (δ=0.02\delta = 0.02) between two models with 80% power in a two-sided paired test at the 5% level. From a pilot run you expect 15% of questions to be discordant. Using n≈(z0.975+z0.8)2 pdisc/δ2n \approx (z_{0.975} + z_{0.8})^2\, p_{\text{disc}} / \delta^2 with z0.975=1.960z_{0.975} = 1.960 and z0.8=0.842z_{0.8} = 0.842, how many questions do you need? Round up to a whole number.

4. The statistics of evaluating AI

CoreHuman

A coding benchmark has 200 problems, each with 8 test cases; the model generates 10 solutions per problem, and a solution counts as correct if it passes all of a problem's test cases. You want a bootstrap confidence interval for pass@1 that reflects uncertainty about which problems were chosen. What should you resample?

Choose one

5. The statistics of evaluating AI

CoreHuman

Which of the following make a reported benchmark score systematically optimistic, rather than just noisy? Select all that apply.

Choose all that apply

6. The statistics of evaluating AI

CoreComputational

Write clustered_accuracy(correct, cluster) returning (accuracy, se) where correct is a 1-D array of 0s and 1s, cluster a same-length array of cluster labels (clusters may have different sizes), and se is the cluster-robust standard error of the mean

SE=1n∑c(∑i∈c(xi−p^))2,\mathrm{SE} = \frac{1}{n}\sqrt{\sum_{c} \Big(\sum_{i \in c} (x_i - \hat p)\Big)^2 } ,

where nn is the number of questions and p^\hat p the overall accuracy.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. The statistics of evaluating AI

StretchComputational

Write ppi_mean(judge_all, judge_lab, human_lab) returning (estimate, se) for the mean human label, by prediction-powered inference. judge_all holds the judge's labels on all NN items; judge_lab and human_lab hold the judge's and the humans' labels on a random subset of nn items. Use

θ^=f‾all+(y−f)‾lab,SE=sf2N+sy−f2n,\hat\theta = \overline{f}_{\text{all}} + \overline{(y - f)}_{\text{lab}}, \qquad \mathrm{SE} = \sqrt{\frac{s^2_f}{N} + \frac{s^2_{y-f}}{n}},

with sample variances computed using ddof=1.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources