Skip to content
LumenaryAcademy
Frontier 75 min

Evaluating models

Benchmarks and their life cycle, contamination, error bars, human and model judges, red-teaming and dangerous-capability evaluations — and how to write an eval worth trusting.

Code tested · 6AI-draftedAwaiting review by a computer scientist

Your guides: Chloe · The Computer Scientist, Margaret · The Experimenter, Gideon · The Verifier

MotivationH

Why evaluation is hard

Every decision about a model depends on measurement: which training run to continue, which checkpoint to release, whether a safety mitigation works, whether a capability has crossed a threshold that demands more caution, whether a research idea helped. If your evaluations are wrong, everything downstream is wrong too, and you won't know.

For a classical machine-learning model the job was simple: one task, a held-out test set, one metric (Learning from data). A frontier language model is evaluated on everything at once, and almost every part of that is hard:

  • Open-ended outputs. Many tasks have no single right answer — an essay, a code review, advice — so grading needs judgement.
  • Breadth. No finite test set covers the space of things people ask.
  • Sensitivity. Scores can move by several points with the prompt format, the number of examples in the prompt, the answer-extraction rule, or the sampling temperature.
  • Contamination. Test questions leak onto the web and into training data.
  • Saturation. Benchmarks that were hard when written are maxed out within a few years.
  • Speed. Models change monthly; careful evaluation takes time and money.
  • Goodhart's law. A measure that becomes a target stops being a good measure. Once a benchmark matters, people optimise for it, deliberately or not.

This lesson is a working researcher's toolkit for these problems. The previous lesson covered agent benchmarks specifically; here we take the general view.

ExplanationH

Benchmark families

Rather than memorise a leaderboard that will be out of date when you read this, learn the families and what each actually measures. Named examples are illustrative.

FamilyWhat it testsTypical format and gradingExamples
KnowledgeBreadth of factual and academic knowledgeMultiple choice, exact matchMMLU (57 subjects) and harder successors
Expert reasoningQuestions experts find hard and that can't be looked up easilyMultiple choice or short answerGPQA, Humanity's Last Exam
MathematicsMulti-step problem solvingFinal answer checked automaticallyGSM8K, competition maths, research-level problem sets
CodingWriting correct programsHidden unit tests; pass@kHumanEval, contest problems released after training cutoffs
AgenticMulti-step work in an environmentEnvironment state checked at the endSWE-bench, web, terminal and desktop tasks
Long contextFinding and using information in long inputsRetrieval and reasoning questions"needle in a haystack" and harder multi-hop variants
Instruction following and chatHelpfulness, format complianceRule checks, human or model judgesinstruction-following suites, pairwise arenas
Safety and honestyRefusals, harmful content, truthfulness, sycophancyClassifiers, judges, behavioural probesmany, often lab-internal

A benchmark has a life cycle. It is published when frontier models score poorly; labs and researchers hill-climb it; scores rise quickly, often faster than its authors expected; and within one to three years the top scores approach the ceiling set by the benchmark's own label noise. MMLU and GSM8K, hard in 2021, were near saturation for frontier models by 2024. The response has been harder benchmarks (expert-written, adversarially filtered against current models), benchmarks with no fixed ceiling (agent tasks of increasing length), and dynamic benchmarks that keep adding fresh questions.

Two metrics deserve a precise definition because they are often misused:

  • pass@k (for code): the probability that at least one of kk sampled solutions passes the tests. Estimating it by drawing exactly kk samples is noisy; the Codex paper gives an unbiased estimator from n≥kn \ge k samples of which cc pass: 1−(n−ck)/(nk)1 - \binom{n-c}{k} / \binom{n}{k}. pass@1 is the number that matters for a user who gets one answer.
  • Win rate and Elo-style ratings (for preferences), discussed below.

Two good habits when reading any reported score: find out exactly how it was computed (prompt, number of shots, sampling settings, how answers were extracted, whether tools were allowed), and check whether the comparison you are making used the same settings. Shared harnesses such as EleutherAI's lm-evaluation-harness, Stanford's HELM and the UK AI Security Institute's Inspect exist largely to make scores reproducible.

ExplanationH

Contamination and saturation

Definition— Contamination

A benchmark is contaminated for a model when its test items (or close paraphrases, or their answers) appeared in the model's training data. Contaminated scores measure memorisation as well as capability, by an unknown amount.

Contamination is hard to avoid. Benchmarks are published on the web, discussed in forums, copied into tutorials and GitHub repositories, and swept into the next crawl (Language models). It is also hard to detect, because most training sets are private. The tools:

  • Decontamination during data processing: remove documents with long nn-gram overlaps with known test sets. This catches verbatim copies, not paraphrases or translations.
  • Canary strings: BIG-bench embedded a unique random string in its files so that data pipelines can filter it, and so that a model's ability to complete the string reveals exposure.
  • Behavioural tests: compare performance on the public test set with performance on a freshly written set of the same difficulty. When Scale AI researchers did this for grade-school maths in 2024, some model families dropped noticeably on the new problems — evidence of overfitting to the public set — while the strongest frontier models showed little drop.
  • Time-based splits: use only problems created after the model's training cutoff, such as contest problems from the last few months. Several dynamic benchmarks refresh their questions continually for this reason.
  • Private held-out sets: keep the test set secret and run models against it yourself. Labs keep internal evaluations private for exactly this reason, and some third-party evaluators keep part of their benchmarks unpublished.

Saturation is the other end of the life cycle. When the best models score 90% or more, the remaining errors are a mix of genuinely hard items and items that are ambiguous or mislabelled; audits of widely used benchmarks have found a non-trivial share of flawed questions. A gain of one point at that stage may be noise, label errors, or fitting the benchmark's quirks. Report it, if at all, with error bars — which brings us to statistics.

ComputationC

Error bars: is model A really better than model B?

A benchmark score is an estimate. The questions are a sample from some larger space of questions you care about, and a different sample would give a different score. Treat accuracy on nn independent questions as a proportion: its standard error is

SE=p^(1−p^)n,\mathrm{SE} = \sqrt{\frac{\hat p (1 - \hat p)}{n}},

and an approximate 95% confidence interval is p^±1.96 SE\hat p \pm 1.96\,\mathrm{SE} (Statistical inference). With n=500n = 500 and p^=0.7\hat p = 0.7, SE≈0.020\mathrm{SE} \approx 0.020: the interval is about ±4\pm 4 points. Many published "improvements" are smaller than that.

The bootstrap gives intervals for any statistic without a formula: resample the questions with replacement many times, recompute the statistic each time, and take the middle 95% of the results. And when comparing two models on the same questions, use a paired analysis — resample questions, not each model's results separately — because much of the variation is shared: hard questions are hard for both.

python
import numpy as np

rng = np.random.default_rng(0)
n = 400
difficulty = rng.normal(0, 1.5, size=n)                   # some questions are hard for everyone
p_a = 1 / (1 + np.exp(-(0.9 - difficulty)))               # model A's chance on each question
p_b = 1 / (1 + np.exp(-(0.7 - difficulty)))               # model B: slightly weaker
a = (rng.random(n) < p_a).astype(float)                   # 1 = answered correctly
b = (rng.random(n) < p_b).astype(float)

def bootstrap(stat, *cols, n_boot=5000, alpha=0.05):
    """Percentile bootstrap: resample questions with replacement, recompute the statistic."""
    idx = rng.integers(0, n, size=(n_boot, n))
    values = np.array([stat(*(c[i] for c in cols)) for i in idx])
    return np.quantile(values, [alpha / 2, 1 - alpha / 2])

acc_a, acc_b = a.mean(), b.mean()
se_a = np.sqrt(acc_a * (1 - acc_a) / n)
print(f"model A accuracy {acc_a:.3f}")
print(f"  normal approximation 95% CI: [{acc_a - 1.96 * se_a:.3f}, {acc_a + 1.96 * se_a:.3f}]")
lo, hi = bootstrap(np.mean, a)
print(f"  bootstrap 95% CI:            [{lo:.3f}, {hi:.3f}]")
print(f"model B accuracy {acc_b:.3f}")

# Is A better than B? Resample the SAME questions for both models (paired) ...
lo, hi = bootstrap(lambda x, y: x.mean() - y.mean(), a, b)
print(f"\ndifference A - B = {acc_a - acc_b:+.3f}")
print(f"  paired bootstrap 95% CI:   [{lo:+.3f}, {hi:+.3f}]")
# ... versus pretending the two results came from independent question sets.
se_diff = np.sqrt(acc_a * (1 - acc_a) / n + acc_b * (1 - acc_b) / n)
print(f"  unpaired normal 95% CI:    [{acc_a - acc_b - 1.96 * se_diff:+.3f}, {acc_a - acc_b + 1.96 * se_diff:+.3f}]")
print(f"questions where the models disagree: {int((a != b).sum())} of {n}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Model A is truly better here (we built it so), and it scores 3.7 points higher. But even the paired interval includes zero: with 400 questions, this evaluation cannot confidently tell the two apart. The paired interval is narrower than the unpaired one because only the 121 questions on which the models disagree carry information about the difference.

Practical rules, many of them set out in Miller (2024):

  • Report intervals, not bare scores, and say how many questions there were.
  • Pair comparisons on the same questions.
  • Cluster when questions are not independent — several questions about the same passage, or the same task with different random seeds — by resampling whole clusters; otherwise intervals are too narrow.
  • Average over samples. With temperature above zero, a model's answer to a question is itself random. Sampling several answers per question and averaging reduces that noise.
  • Beware many comparisons. Test twenty variants and one will look significantly better by chance. Hold out a final test set you only look at once.
  • Plan the size. To estimate an accuracy near 50% to within ±E\pm E with 95% confidence, you need about n≈1.962⋅0.25/E2n \approx 1.96^2 \cdot 0.25 / E^2 independent questions: about 2,400 for ±2\pm 2 points.
ExplanationH

Human preference evaluation

For open-ended chat, the most direct measure is to ask people which response they prefer. The best-known public example is LMArena (formerly Chatbot Arena, started by researchers at UC Berkeley): anyone can type a prompt, receive two anonymous responses from different models, and vote. Millions of votes are turned into a rating per model with the Bradley–Terry model — the same one used for reward models in Post-training — where P(i beats j)=σ(si−sj)P(i \text{ beats } j) = \sigma(s_i - s_j), reported on a chess-like Elo scale (Chiang et al., 2024).

Its strengths are real: prompts from real users, no fixed test set to memorise, and a single number that tracks something people care about. So are its biases:

  • Style over substance. Voters, like judges of student essays, favour longer answers, more formatting (headings, bullet lists, bold text) and confident tone, independent of correctness.
  • Unverified correctness. A casual voter often can't tell whether a medical, legal or mathematical answer is right.
  • Sycophancy. People like being agreed with; optimising for votes rewards flattery.
  • Prompt distribution. The prompts are whatever visitors type, which may not resemble your use.
  • Gaming. A 2025 critique ("The Leaderboard Illusion") argued that some providers tested many private variants and published only the best, and that sampling and data access were uneven across providers; the arena's operators disputed parts of the analysis.

A partial remedy for the first bias is style control: fit the Bradley–Terry model with extra features for response length and formatting, so that their effect is estimated separately from each model's strength. Here is the idea on simulated votes, where one model is slightly weaker but much more verbose:

python
import numpy as np

rng = np.random.default_rng(2)
models = ["Atlas", "Baobab", "Condor", "Delta"]
skill = np.array([0.0, 0.3, 0.5, 0.4])        # true quality (unknown to the leaderboard)
verbosity = np.array([0.0, 0.0, 0.0, 1.5])    # Delta writes much longer answers

# Simulated arena: random pairs, and voters who reward length as well as quality.
m = 8000
i = rng.integers(0, 4, m)
j = (i + rng.integers(1, 4, m)) % 4                       # a different model
len_diff = (verbosity[i] - verbosity[j]) + rng.normal(0, 0.5, m)
p_i_wins = 1 / (1 + np.exp(-(skill[i] - skill[j] + 0.5 * len_diff)))
y = (rng.random(m) < p_i_wins).astype(float)

def fit_logistic(X, y, steps=25):
    """Maximum-likelihood logistic regression by Newton's method."""
    w = np.zeros(X.shape[1])
    for _ in range(steps):
        p = 1 / (1 + np.exp(-X @ w))
        H = X.T @ (X * (p * (1 - p))[:, None]) + 1e-6 * np.eye(len(w))
        w += np.linalg.solve(H, X.T @ (y - p))
    return w

# Bradley-Terry: one strength per model (Atlas fixed at 0), P(i beats j) = sigmoid(s_i - s_j).
X = np.zeros((m, 4)); X[np.arange(m), i] += 1; X[np.arange(m), j] -= 1
X = X[:, 1:]
plain = np.concatenate([[0.0], fit_logistic(X, y)])
# "Style control": add the length difference as an extra feature, so it gets its own coefficient.
w = fit_logistic(np.column_stack([X, len_diff]), y)
controlled = np.concatenate([[0.0], w[:3]])

def to_elo(s):                                             # the familiar arena scale
    return 1000 + 400 / np.log(10) * s

print(f"{'model':8s} {'true':>6s} {'plain BT':>9s} {'style-controlled':>17s}")
for k in np.argsort(-skill):
    print(f"{models[k]:8s} {to_elo(skill[k]):6.0f} {to_elo(plain[k]):9.0f} {to_elo(controlled[k]):17.0f}")
print(f"\nfitted length coefficient: {w[3]:.2f} (true 0.50)")
print("plain ranking:          ", [models[k] for k in np.argsort(-plain)])
print("style-controlled ranking:", [models[k] for k in np.argsort(-controlled)])
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The plain leaderboard puts verbose Delta first by more than a hundred points; controlling for length recovers the true order and the true size of the length effect. Real style control is less clean — length is only one of many style features, and they correlate with genuine quality — but the principle is sound regression practice: estimate the confounder rather than letting it hide in the ranking.

For specialised domains, labs and evaluators also run expert preference studies — doctors grading medical answers, lawyers grading legal ones — which are slower and costlier but measure correctness as well as appeal.

ComputationC

LLM-as-judge

Human judgement is slow and expensive, so much evaluation now uses a strong model as the judge: given a question, one or two responses and a rubric, it outputs a score or a preference. Zheng et al. (2023) found that a strong judge agreed with human experts about as often as experts agreed with each other on their chat benchmark. That made LLM judges standard, and they are genuinely useful. They also have well-documented biases (Wang et al., 2023; Zheng et al.):

  • Position bias: preferring whichever response is shown first (or second).
  • Verbosity bias: preferring longer responses.
  • Self-preference: rating outputs of their own model family more highly.
  • Limited competence: a judge that can't solve the problem can't reliably grade the solution.
  • Exploitability: if a policy is trained against a judge, it learns to please the judge (Goodhart again).

Here is a simulated judge with position and length biases, and the standard first mitigation, swapping the order:

python
import numpy as np

rng = np.random.default_rng(1)
n = 2000
quality = rng.normal(size=(n, 2))            # true quality of response X and response Y in each pair
length = rng.normal(size=(n, 2))             # length, unrelated to quality
truth = quality[:, 0] > quality[:, 1]        # is X really better?

def judge_prefers_first(q_first, q_second, len_first, len_second):
    """A simulated LLM judge: tracks quality, but likes whatever is shown first, and longer answers."""
    logit = 2.0 * (q_first - q_second) + 0.8 + 0.7 * (len_first - len_second)
    return rng.random(len(q_first)) < 1 / (1 + np.exp(-logit))

x_first = judge_prefers_first(quality[:, 0], quality[:, 1], length[:, 0], length[:, 1])    # order X, Y
y_first = judge_prefers_first(quality[:, 1], quality[:, 0], length[:, 1], length[:, 0])    # order Y, X
x_wins_ab = x_first                       # verdict for X with X shown first
x_wins_ba = ~y_first                      # verdict for X with Y shown first

print(f"X shown first:  judge picks X {x_wins_ab.mean():.0%} of the time, agrees with truth {np.mean(x_wins_ab == truth):.0%}")
print(f"Y shown first:  judge picks X {x_wins_ba.mean():.0%} of the time, agrees with truth {np.mean(x_wins_ba == truth):.0%}")
consistent = x_wins_ab == x_wins_ba
print(f"verdict flips when the order is swapped: {1 - consistent.mean():.0%} of pairs")
print(f"keeping only order-consistent verdicts: agrees with truth {np.mean(x_wins_ab[consistent] == truth[consistent]):.0%}"
      f" on {consistent.mean():.0%} of pairs")

# Swapping cancels position bias but not length bias:
longer_x = length[:, 0] > length[:, 1]
both = consistent & (x_wins_ab != truth)                  # consistent but wrong
print(f"among consistent-but-wrong verdicts, the judge chose the longer answer {np.mean(x_wins_ab[both] == longer_x[both]):.0%} of the time")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Swapping exposes the position bias — a quarter of verdicts flip — and keeping only consistent verdicts (or counting inconsistent ones as ties) raises agreement with the truth. It does nothing for length bias: the remaining errors overwhelmingly favour the longer answer. Each bias needs its own countermeasure. The standard toolkit:

  • run both orders and treat disagreement as a tie;
  • grade against a rubric and, where possible, a reference answer, scoring specific criteria rather than overall preference;
  • ask for brief reasoning before the verdict, and constrain the output format;
  • use a judge from a different model family from the models being judged, or several judges;
  • calibrate: have humans label a random sample, and report the judge's agreement with them (with a chance-corrected statistic such as Cohen's kappa) alongside the results;
  • never train a model against the same judge you use to evaluate it.
ExplanationH

Red-teaming and dangerous-capability evaluations

Benchmarks ask "how good is the model at X?". Two other kinds of evaluation ask "how can this go wrong?".

Red-teaming is adversarial testing: people (and increasingly models) try to make the system produce harmful, false or policy-violating outputs, or to take harmful actions. Anthropic's Ganguli et al. (2022) described a large human red-teaming effort and released the data; Perez et al. (2022) used one language model to generate attacks on another, which scales the search, and related work used models to write whole behavioural evaluations (Perez et al.). Good red-teaming is structured by a risk taxonomy, involves domain experts for specialised harms, and feeds back into training and safeguards. Labs also run external red-teaming programmes and bug bounties.

Dangerous-capability evaluations ask whether a model has capabilities that could enable severe harm if misused or if the model itself behaved badly (Shevlane et al., 2023). Common domains as of 2026:

  • uplift for biological, chemical, radiological or nuclear weapons (does the model meaningfully help a novice beyond what the internet already provides?), usually tested with experts and controlled studies;
  • offensive cyber capability, often measured with capture-the-flag challenges and vulnerability-discovery tasks;
  • autonomy and AI research capability: can the model carry out long software and machine-learning research tasks on its own, acquire resources, or replicate itself?
  • persuasion and manipulation;
  • deception and scheming in agentic settings: does a model, when its goals conflict with its instructions, covertly work around them? (AI safety and alignment.)

Results of these evaluations feed the labs' frontier safety frameworks, which tie capability thresholds to required safeguards, and are summarised in the system or model cards that major labs publish with releases. Government bodies such as the UK's AI Security Institute have carried out pre-deployment testing of frontier models in cooperation with developers.

The central methodological issue is elicitation. A dangerous-capability evaluation is trying to establish an upper bound on what the model can do. If the evaluators use a weak prompt, no tools, too few attempts or too little time, the model may look less capable than it is, and a determined user with better scaffolding could get more out of it. A negative result is only as strong as the effort that went into trying to get a positive one. Good practice includes giving the model realistic tools, fine-tuning it for the task where feasible, trying many attempts, and reporting the elicitation effort with the result. A newer complication: some models show signs of recognising when they are being evaluated, which may change their behaviour — an active area of study.

Worked examplesH

Writing a good eval

Most of the useful evaluations in the world are not famous benchmarks but small, specific ones written by a team for its own product or research question. Writing one well is a core skill.

Example— Designing an eval for a support assistant

Suppose you are building a model-based assistant for a water utility's customer service in Nairobi and Kampala, answering in English and Swahili.

  1. Define the construct. What exactly should the eval measure? "Answers billing questions correctly, in the customer's language, following the refund policy, and escalates to a human when the customer reports a burst pipe." Vague goals ("is helpful") produce vague evals.
  2. Sample realistic inputs. Draw questions from real (anonymised, consented) support logs, in both languages and in proportions that match real traffic, plus a deliberately chosen set of rare but important cases (emergencies, angry customers, policy edge cases).
  3. Make grading unambiguous. For each item write the reference facts and the pass criteria. Prefer checks that a program can apply (did it call the escalation tool? does the amount match?) and use a rubric-guided model judge only for what can't be checked, calibrated against staff labels.
  4. Validate the items. Have two people answer and grade each item independently; fix or drop items they disagree on. Many published benchmarks skipped this step and it shows.
  5. Set baselines. The current system, a simple retrieval-only system, and staff performance.
  6. Size it. For ±3\pm 3 points at 95% confidence you need roughly a thousand independent items for the headline number; rare categories need enough items of their own to say anything.
  7. Keep it private and versioned. Don't paste it into prompts or public repositories; track changes; keep a final held-out portion.
  8. Report honestly. Intervals, per-category results, failure examples, and the exact settings used.

Two qualities to keep in mind are validity (does a high score mean what you think it means?) and reliability (would you get the same score again?). Most evaluation failures are validity failures: an eval that is easy to score well on for the wrong reasons. The best defence is to read many model outputs yourself, especially the ones that passed.

Common misconceptionsH

Common misconceptions

Misconception

"A one-point gain on a benchmark is progress." On a few hundred questions, a point is usually within noise, and on a saturated benchmark it may reflect label errors or contamination. Ask for the interval.

Misconception

"A benchmark score measures a capability." It measures performance on particular items under particular settings. Whether that generalises to the capability you care about — construct validity — is a separate question that needs its own evidence.

Misconception

"LLM judges are objective because they are machines." They have systematic, measurable biases toward position, length and their own style, and they can be gamed. Calibrate them against humans and report agreement.

Misconception

"The top of the arena leaderboard is the best model for my task." It is the model that crowd voters preferred on crowd prompts. Your task, users and criteria may differ; test on your own distribution.

Misconception

"The evaluation found no dangerous capability, so the model is safe." Absence of evidence under weak elicitation is weak evidence of absence. The result is only as strong as the effort made to elicit the capability.

Exercises

Exercises

1. Evaluating models

CoreComputational

Write bootstrap_ci(correct, n_boot=2000, alpha=0.05, seed=0). correct is a 1-D numpy array of 0s and 1s, one per question. Using rng = np.random.default_rng(seed), draw n_boot resamples of the questions with replacement (each the same size as correct), compute the accuracy of each, and return the (alpha/2, 1 - alpha/2) quantiles of those accuracies as a tuple of two floats.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

2. Evaluating models

CoreHuman

You want a 95% confidence interval of ±3\pm 3 percentage points on an accuracy that could be anywhere, so plan for the worst case p=0.5p = 0.5. Using n=1.962 p(1−p)/E2n = 1.96^2\,p(1-p)/E^2, how many independent questions do you need? Round up to a whole number.

3. Evaluating models

CoreComputational

Write pass_at_k(n, c, k): the unbiased estimator of pass@k from nn samples of which cc are correct,

1−(n−ck)/(nk).1 - \binom{n - c}{k} \Big/ \binom{n}{k}.

If n−c<kn - c < k, every set of kk samples contains a correct one, so return 1.0. Return a float.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

4. Evaluating models

CoreHuman

A model scores 92% on a public maths benchmark. Which finding would be the strongest evidence that the score is inflated by contamination?

Choose one

5. Evaluating models

CoreHuman

You use an LLM judge to compare two chat models. Which practices reduce known judge biases or make them measurable? Select all that apply.

Choose all that apply

6. Evaluating models

Warm-upHuman

A model answers 320 of 400 independent questions correctly. Using the normal approximation, what is the half-width of the 95% confidence interval, in percentage points, rounded to the nearest whole number?

7. Evaluating models

CoreHuman

A dangerous-capability evaluation of an offensive-cyber task finds the model succeeds 0 times in 20 attempts. The evaluators used a single short prompt, no tools, and no retries. What is the right conclusion?

Choose one

8. Evaluating models

StretchComputational

Write cohens_kappa(a, b) measuring agreement between a judge and a human beyond chance. a and b are equal-length lists of labels (any hashable values). With pop_o the fraction of items where they agree and pe=∑ℓPa(ℓ) Pb(ℓ)p_e = \sum_\ell P_a(\ell)\,P_b(\ell) the agreement expected by chance from each rater's own label frequencies, return κ=(po−pe)/(1−pe)\kappa = (p_o - p_e)/(1 - p_e). If pe=1p_e = 1, return 1.0.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources