How long until?
A trial in several hospitals tests whether a new maintenance drug delays relapse in patients recovering from a serious illness. Patients join over two years and the trial closes after three. When it closes, some patients have relapsed, at known times. Many have not: one joined eight months ago and is well; another was well at her last visit before she moved to another city. How long, on average, until relapse?
A water agency in a dry region faces the same question about hand pumps in villages: how long until a pump fails, and does a new seal design make it last longer? When the study ends, many pumps are still working.
These are time-to-event questions, and their data have three features that defeat the methods of earlier lessons. Times are positive and usually skewed, with a long right tail. The interesting quantity is often not the mean but the chance of lasting past a certain time. And, decisively, many observations are incomplete: for the patient who has been well for eight months, we know only that her time to relapse is more than eight months. Throwing such observations away, or pretending they are relapses, gives badly wrong answers. Survival analysis is the set of methods built to use them correctly. The name comes from its origin in mortality tables, but the "event" can be anything that happens once: relapse, machine failure, finding a job, a customer cancelling a subscription, a student completing a degree.
Censoring, and why it matters
An observation is right-censored when we know only that the event had not happened by some time. In the trial, censoring happens for two reasons: administrative censoring when the study closes, and loss to follow-up when a patient moves away or stops attending. Because patients join at different calendar times, the study's closing date cuts their follow-up at different lengths, as the figure shows.
Each patient contributes a pair: a time and an indicator , equal to 1 if the event was observed at and 0 if the observation was censored then. Other kinds of incompleteness exist (left censoring, when the event happened before observation began, and interval censoring, when we know only that it happened between two check-ups), but right censoring is by far the most common, and this lesson deals with it.
The methods below rest on one assumption that deserves to be stated plainly.
Definition— non-informative censoring
Censoring is non-informative (or independent) if, among people with the same covariates, those censored at time have the same future risk of the event as those who remain under observation. Being censored must tell us nothing about one's prognosis.
Administrative censoring at the close of a study usually satisfies this. Loss to follow-up may not: if patients stop attending because they feel too ill to travel, the people left in the study are healthier than those who left, and every method in this lesson becomes optimistic.
How badly do the naive approaches fail? The simulation below creates trials of 200 patients with a true median time to relapse of 20 months, staggered entry over two years, closure at month 36 and some random loss to follow-up, and estimates the median three ways.
import numpy as np
rng = np.random.default_rng(1958)
true_median = 20.0 # months; exponential time to relapse
rate = np.log(2) / true_median
def km_median(time, event):
"""Smallest event time at which the Kaplan-Meier curve reaches 0.5 or below (nan if it never does)."""
order = np.lexsort((1 - event, time)) # at tied times, events before censorings
time, event = time[order], event[order]
at_risk = len(time) - np.arange(len(time))
s = np.cumprod(1 - event / at_risk)
hit = np.nonzero(s <= 0.5)[0]
return time[hit[0]] if hit.size else np.nan
results = {"KM median": [], "median, censored treated as relapses": [], "median of relapses only": []}
share_censored = []
for rep in range(2000):
n = 200
entry = rng.uniform(0, 24, n) # patients join over two years
t_relapse = rng.exponential(1 / rate, n)
t_dropout = rng.exponential(60, n) # some move away, unrelated to their health
follow = np.minimum(36 - entry, t_dropout) # the study closes at month 36
time = np.minimum(t_relapse, follow)
event = (t_relapse <= follow).astype(float)
share_censored.append(1 - event.mean())
results["KM median"].append(km_median(time, event))
results["median, censored treated as relapses"].append(np.median(time))
results["median of relapses only"].append(np.median(time[event == 1]))
print(f"true median {true_median:.1f} months; on average {np.mean(share_censored):.0%} of patients are censored")
for name, est in results.items():
print(f"{name:38} average {np.nanmean(est):5.1f} middle 95% of estimates {np.nanpercentile(est, 2.5):5.1f} to {np.nanpercentile(est, 97.5):5.1f}")More than half the patients are censored in a typical trial (54%). Treating their censoring times as relapse times gives an average median of 12.8 months; keeping only the patients who relapsed gives 8.3, because a relapse is more likely to be observed when it comes early. Both are far below 20, and both are confidently wrong: their middle 95% of estimates never come near the truth. The Kaplan–Meier estimate, introduced below, averages 20.3. Its spread (16.0 to 26.1) is wider, which is the honest price of having less information.
Survival, hazard and cumulative hazard
Let be the time to the event, with density and distribution function .
Definition— survival function and hazard
The survival function is , the probability of lasting beyond . The hazard function is the event rate at time among those who have lasted that long, and the cumulative hazard is . They are linked by .
The last identity follows because ; integrating from 0 to with gives . So any one of , and determines the others (see calculus of one variable for the integration).
The hazard is the most natural way to think about time to an event, because its shape tells a story. A constant hazard gives the exponential distribution, , with median and mean : failure by random shocks, with no ageing. A Weibull distribution, , has hazard proportional to : falling if (early failures from manufacturing defects, or risk just after surgery), constant if , rising if (wear-out). Engineers know the combination as the bathtub curve, and human mortality has the same shape over a lifetime: high risk in the first year of life, a long low stretch, then risk rising steadily with age.
The Kaplan–Meier estimator
Edward Kaplan and Paul Meier's 1958 idea is to break survival into a chain of conditional probabilities, one for each time an event happens. To survive past month 5, a patient must survive month 2, then (given that) month 3, then (given that) month 5. Each conditional probability is estimated from the people still at risk at that moment, the ones neither failed nor censored yet, and censored patients count as at risk for as long as they were observed.
Definition— Kaplan–Meier estimator
Let be the distinct times at which events occur, the number of events at , and the number at risk just before . Then a step function that drops only at event times. Greenwood's formula estimates its variance: .
Example— Twelve patients by hand
Twelve patients' months to relapse, with censored times marked +: 2, 3, 3+, 5, 6+, 7, 9, 10+, 12, 14+, 15, 18+. At month 2 all 12 are at risk and one relapses: . At month 3, 11 are at risk and one relapses: . The patient censored at month 3 leaves the risk set after that month's events (the usual convention for ties), as does no one else until month 5, so 9 are at risk there: . The censored patient at month 6 never causes a drop; she simply shrinks the risk set for later months.
The code below finishes the table, adds Greenwood standard errors, and checks the result against SciPy's implementation.
import numpy as np
from scipy import stats
# Months from starting treatment to relapse (event = 1) or to the last check-up without relapse (event = 0).
time = np.array([2, 3, 3, 5, 6, 7, 9, 10, 12, 14, 15, 18])
event = np.array([1, 1, 0, 1, 0, 1, 1, 0, 1, 0, 1, 0])
def kaplan_meier(time, event):
"""Return the distinct event times, numbers at risk, events, S(t) and Greenwood standard errors."""
rows, s, green = [], 1.0, 0.0
for t in np.unique(time[event == 1]):
n = np.sum(time >= t) # at risk just before t
d = np.sum((time == t) & (event == 1)) # events at t
s *= 1 - d / n
green += d / (n * (n - d)) if n > d else 0.0
rows.append((t, n, d, s, s * np.sqrt(green)))
return rows
print(" month at risk relapses S(t) std. error")
for t, n, d, s, se in kaplan_meier(time, event):
print(f"{t:5d} {n:7d} {d:8d} {s:.3f} {se:.3f}")
# The same estimate from SciPy (version 1.11 or later), which handles censoring through CensoredData.
data = stats.CensoredData(uncensored=time[event == 1], right=time[event == 0])
res = stats.ecdf(data)
print("SciPy S(t) at the same months:", np.round(res.sf.evaluate(np.unique(time[event == 1])), 3))
print(f"naive: mean of all 12 times = {time.mean():.1f} months; share relapsed = {event.mean():.0%}")The estimated median, the first time reaches 0.5 or below, is 12 months (). Notice how the standard error grows along the curve, from 0.080 at month 2 to 0.163 at month 15, as the risk set shrinks to two patients: the right-hand end of any Kaplan–Meier curve rests on very few people, and large late drops (here from 0.397 to 0.198 at a single relapse) should be read with that in mind. Good plots print the number at risk beneath the time axis for exactly this reason.
Comparing groups: the log-rank test
Is the new pump seal better? Plot a Kaplan–Meier curve for each group, then test whether the curves differ. The log-rank test asks, at each event time: given how many units of each group were at risk and how many failed in total, how many failures would we expect in group B if the groups had the same hazard? It adds up observed minus expected over all event times and divides by the standard deviation of that sum. Under the null hypothesis of equal hazards, the squared standardised sum is approximately chi-square with one degree of freedom.
import numpy as np
from scipy import stats
rng = np.random.default_rng(31)
# Village hand pumps fitted with an old (A) or new (B) seal design, followed for up to 5 years.
# Failure times are Weibull with shape 1.5 (wear-out); design B has 0.6 times the hazard of A.
n = 80
shape, scale_a = 1.5, 4.0
scale_b = scale_a * 0.6 ** (-1 / shape) # proportional hazards with ratio 0.6
fail = np.r_[scale_a * rng.weibull(shape, n), scale_b * rng.weibull(shape, n)]
group = np.r_[np.zeros(n), np.ones(n)]
installed = rng.uniform(0, 2, 2 * n) # installed over the first two years
follow = 5 - installed # observation ends after year 5
time = np.minimum(fail, follow)
event = (fail <= follow).astype(int)
def log_rank(time, event, group):
o_minus_e, var = 0.0, 0.0
for t in np.unique(time[event == 1]):
at_risk = time >= t
n_all, n_b = at_risk.sum(), (at_risk & (group == 1)).sum()
d_all = ((time == t) & (event == 1)).sum()
d_b = ((time == t) & (event == 1) & (group == 1)).sum()
o_minus_e += d_b - d_all * n_b / n_all # observed minus expected failures in group B
if n_all > 1:
var += d_all * (n_b / n_all) * (1 - n_b / n_all) * (n_all - d_all) / (n_all - 1)
return o_minus_e, o_minus_e ** 2 / var
for g, name in [(0, "A (old seal)"), (1, "B (new seal)")]:
m = group == g
print(f"{name}: {event[m].sum()} failures, {(1 - event[m]).sum()} still working at the end of observation")
o_e, chi2 = log_rank(time, event, group)
print(f"group B: observed minus expected failures = {o_e:.1f}; log-rank chi-square = {chi2:.2f}, "
f"p = {stats.chi2.sf(chi2, df=1):.4f}")
res = stats.logrank(stats.CensoredData(uncensored=time[(group == 0) & (event == 1)], right=time[(group == 0) & (event == 0)]),
stats.CensoredData(uncensored=time[(group == 1) & (event == 1)], right=time[(group == 1) & (event == 0)]))
print(f"SciPy's logrank: statistic {res.statistic:.3f} (its square is {res.statistic ** 2:.2f}), p = {res.pvalue:.4f}")Pumps with the new seal had 10.0 fewer failures than expected under equal hazards; the log-rank chi-square is 4.34 with . SciPy reports the same test as a standard normal statistic, 2.084, whose square is the chi-square. The curves below show where the difference comes from: they run close together for the first year and separate steadily after it.
The log-rank test gives every event time equal weight, which makes it most powerful when one group's hazard is a constant multiple of the other's. If the curves cross, as when surgery carries an early risk but a later benefit compared with medication, early and late differences cancel in the sum, and the test can report a large -value for two very different treatments. Weighted versions emphasise early or late differences, but the weighting must be chosen before seeing the data.
Hazard ratios and the Cox model
A test says whether groups differ; we also want how much, and to adjust for other variables such as age. In 1972 David Cox proposed modelling the hazard of unit with covariates as where is a baseline hazard left completely unspecified. Each covariate multiplies the hazard by a constant factor at every time: is the hazard ratio for a one-unit increase in , holding the others fixed. Because the ratio is the same at all times, this is the proportional hazards model.
Cox's insight was that can be estimated without knowing at all. At each event time, ask: given that someone in the risk set failed now, what is the probability that it was the one who did? Under the model, it is that unit's share of the risk set's total hazard, and cancels: where is the set at risk at time . Maximising this partial likelihood gives estimates that behave like maximum-likelihood estimates, with standard errors from its curvature (Likelihood and model selection). The code fits it directly on two simulated trials with very different baseline hazards, one falling and one rising; the model does not need to be told which.
import numpy as np
from scipy.optimize import minimize
rng = np.random.default_rng(1972)
def simulate(n, shape):
"""Randomised treatment (hazard ratio 0.7) and age (hazard ratio 1.35 per 10 years) on top of a
Weibull baseline hazard with the given shape; censoring at a uniform time up to 6 years."""
treat = rng.integers(0, 2, n)
age10 = rng.normal(0, 1, n) # age in decades, centred at the mean age
eta = np.log(0.7) * treat + np.log(1.35) * age10
t_event = (rng.exponential(1, n) / np.exp(eta)) ** (1 / shape) * 3.0
t_cens = rng.uniform(0, 6, n)
return np.minimum(t_event, t_cens), (t_event <= t_cens).astype(int), np.column_stack([treat, age10])
def neg_log_partial_likelihood(beta, time, event, X):
order = np.argsort(-time) # longest times first, so cumulative sums give risk sets
eta = X[order] @ beta
log_risk = np.logaddexp.accumulate(eta) # log of the sum of exp(eta) over everyone still at risk
return -np.sum((eta - log_risk)[event[order] == 1])
def hessian(f, x, h=1e-4):
"""Central-difference second derivatives of f at x."""
k = len(x)
H = np.zeros((k, k))
for i in range(k):
for j in range(k):
ei, ej = np.eye(k)[i] * h, np.eye(k)[j] * h
H[i, j] = (f(x + ei + ej) - f(x + ei - ej) - f(x - ei + ej) + f(x - ei - ej)) / (4 * h * h)
return H
for shape, story in [(0.7, "falling hazard (early risk)"), (2.5, "rising hazard (wear-out)")]:
time, event, X = simulate(1500, shape)
fit = minimize(neg_log_partial_likelihood, np.zeros(2), args=(time, event, X), method="BFGS")
se = np.sqrt(np.diag(np.linalg.inv(hessian(lambda b: neg_log_partial_likelihood(b, time, event, X), fit.x))))
lo, hi = np.exp(fit.x - 1.96 * se), np.exp(fit.x + 1.96 * se)
print(f"{story}: {event.sum()} events")
print(f" treatment: hazard ratio {np.exp(fit.x[0]):.2f}, 95% CI {lo[0]:.2f} to {hi[0]:.2f} (true 0.70)")
print(f" age, per decade: hazard ratio {np.exp(fit.x[1]):.2f}, 95% CI {lo[1]:.2f} to {hi[1]:.2f} (true 1.35)")In both trials the estimates land near the truth (treatment 0.66 and 0.73, against 0.70), and every 95% interval contains the true value, although one baseline hazard falls with time and the other rises steeply. That robustness to the baseline is why the Cox model became the standard regression model of clinical research; Cox's paper is one of the most cited in all of statistics.
The price is the proportional-hazards assumption itself. It is checked by plotting for each group, which gives parallel curves under the model, or with tests based on Schoenfeld residuals (Grambsch and Therneau, 1994). When it fails, options include stratifying the baseline hazard by the offending variable, letting a coefficient change with time, or switching summaries.
Survival analysis in medicine and engineering
Medicine. Survival methods are the core of clinical trials with time-to-event endpoints (overall survival, progression-free survival, time to hospital readmission) and of cancer registries, which publish survival at one, five and ten years by cancer type and stage, such as the US National Cancer Institute's SEER programme. Two complications recur. Competing risks: in a study of time to kidney failure among elderly patients, death from heart disease prevents kidney failure from ever being observed, and treating such deaths as ordinary censoring overstates the risk of kidney failure; the cumulative incidence function and the Fine–Gray model handle this. And trial data concern real patients, so they are analysed under consent, ethics approval and data-protection law; shared data sets are de-identified, and the lesson's examples are simulated.
Engineering reliability. Reliability engineers call the same ideas life data analysis. Failure times of bearings, pumps, turbine blades and electronic boards are fitted with Weibull models (the Swedish engineer Waloddi Weibull proposed the distribution in 1951), and the fitted shape parameter is read directly: suggests manufacturing defects worth screening out by "burn-in"; suggests wear-out, for which preventive replacement before the hazard climbs makes sense. Test units still running when a test stops are censored, exactly like patients still well at the end of a trial. Accelerated life tests run units at higher temperature or load to provoke failures sooner, then extrapolate to normal use with a physical model; the extrapolation is only as good as that model. The NIST e-Handbook covers these methods in detail.
The same mathematics measures unemployment duration in labour economics, customer churn in subscription businesses, time to degree in education research and the timing of first marriage in demography.
From life tables to software
John Graunt's analysis of London's bills of mortality in 1662 and Edmond Halley's life table for the city of Breslau in 1693 are the ancestors of survival analysis; actuaries used life tables to price annuities for two centuries before statisticians formalised censoring. Kaplan and Meier's 1958 paper (written after the two discovered they had submitted similar work), the log-rank test of Nathan Mantel and of Richard and Julian Peto in the 1960s and early 1970s, and Cox's 1972 model created the modern subject, and counting-process theory in the 1970s and 1980s, led by Odd Aalen and others, gave it rigorous foundations.
In practice, the R survival package by Terry Therneau is the reference implementation; in Python, lifelines and scikit-survival provide Kaplan–Meier, Cox and parametric models, and SciPy has Kaplan–Meier (stats.ecdf) and the log-rank test. A typical R analysis looks like this (not run here):
# R, not run here
library(survival)
fit_km <- survfit(Surv(months, relapsed) ~ arm, data = trial)
survdiff(Surv(months, relapsed) ~ arm, data = trial) # log-rank test
fit_cox <- coxph(Surv(months, relapsed) ~ arm + age, data = trial)
cox.zph(fit_cox) # check proportional hazardsMachine learning has added flexible models such as random survival forests and neural networks that predict individual survival curves; they still need censoring-aware loss functions and censoring-aware evaluation, such as the concordance index, and calibration checked against Kaplan–Meier estimates.
Common misconceptions
Misconception
"Censored observations are missing data, so drop them." They carry information (the event had not happened by time ), and dropping them keeps only those whose events came early. In the simulation, the median of relapses only was 8.3 months against a truth of 20.
Misconception
"Kaplan–Meier works whatever the reason for censoring." It assumes censored units have the same future risk as those still observed. If the sickest patients drop out, the curve is too optimistic, and no amount of data fixes it. Ask why each observation was censored.
Misconception
"A hazard ratio of 0.5 means patients live twice as long." It means that, at each moment, a treated patient still at risk has half the event rate of a comparable untreated one. How that translates into extra time depends on the baseline hazard: for an exponential distribution the median doubles, but for other shapes it does not, and a hazard ratio alone says nothing about absolute benefit. Report survival probabilities or restricted mean survival alongside it.
Misconception
"The proportion who had the event is the risk of the event." Fifty-eight per cent of the twelve patients in the example relapsed, but the Kaplan–Meier estimate of relapse by month 15 is . The raw proportion depends on how long each person happened to be followed.
Exercises
1. Survival analysis
Warm-upHumanFive patients in a small study in Hanoi are followed after surgery. Their months to recurrence, with censored times marked +, are: 3, 5+, 6, 9, 12+. What is the Kaplan–Meier estimate , as a percentage rounded to the nearest whole number?
2. Survival analysis
CoreHumanIn a study of a home-based treatment for heart failure, patients who feel too unwell to attend clinic visits often stop attending and are recorded as censored at their last visit. What does this do to the Kaplan–Meier estimate of survival?
3. Survival analysis
CoreHumanPumps in a reliability study fail at a constant hazard of 0.02 per month (an exponential model). What is the median time to failure, to the nearest month?
4. Survival analysis
CoreHumanA randomised trial reports a hazard ratio for death of 0.6 (95% CI 0.48 to 0.75) for a new drug against standard care, from a Cox model. Which statement is correct?
5. Survival analysis
StretchHumanTwo treatments for a heart condition are compared: surgery has a higher risk of death in the first months but a lower risk afterwards; medication has a steady risk. The Kaplan–Meier curves cross at about two years. The log-rank test gives . What is the best conclusion?
6. Survival analysis
CoreComputationalWrite km_at(time, event, t), which returns the Kaplan–Meier estimate as a float. time holds each unit's observed time and event is 1 for an observed event and 0 for a censored observation.
At each distinct event time , multiply by , where is the number of events at and the number of units with observed time (so a unit censored at exactly still counts as at risk at ).
7. Survival analysis
CoreComputationalFor an exponential model with rate , a unit that fails at time contributes to the likelihood and a unit censored at contributes . So the log-likelihood is , where is the number of failures.
Write exp_rate_mle(time, event) that returns the maximum-likelihood estimate of as a float. (Set the derivative of to zero.)
Further reading and resources
- Nonparametric estimation from incomplete observations · Edward L. Kaplan and Paul Meier · 1958
- Regression models and life-tables · David R. Cox · 1972
- Proportional hazards tests and diagnostics based on weighted residuals · Patricia M. Grambsch and Terry M. Therneau · 1994
- The hazards of hazard ratios · Miguel A. Hernán · 2010
- Restricted mean survival time: an alternative to the hazard ratio for the design and analysis of randomized trials with a time-to-event outcome · Patrick Royston and Mahesh K. B. Parmar · 2013
- A proportional hazards model for the subdistribution of a competing risk · Jason P. Fine and Robert J. Gray · 1999
- NIST/SEMATECH e-Handbook: Assessing product reliability · National Institute of Standards and TechnologyHazard shapes, the Weibull distribution, censored reliability data and accelerated life testing.
- survival: the R package · Terry M. Therneau
- lifelinesA Python library for survival analysis.
- scikit-survivalSurvival models in the scikit-learn style, including random survival forests.
- SciPy: scipy.stats.ecdf and scipy.stats.logrank