Skip to content
LumenaryAcademy
Foundations 70 min

Comparing groups

Welch's and paired t-tests, proportions and chi-squared tests, rank and permutation tests — and choosing the test by the design of the study, not by trying them all.

Code tested · 6AI-draftedAwaiting review by a statistician

Your guides: Chloe · The Statistician, Margaret · The Simulator, Gideon · The Sceptic

MotivationH

Is the difference real?

Most questions that reach a statistician are comparisons. Does an improved sorghum variety yield more than the local one? Do runners go faster in a new shoe? Do SMS reminders bring more children back for their second vaccine dose? Do patients stay longer on one hospital ward than another? In each case two groups differ in the data, and the question is whether the difference reflects something real or just the luck of who ended up in which group.

Hypothesis testing gave the logic: assume no real difference, ask how surprising the data would be, and report the size of the difference with its interval. This lesson supplies the tools for the commonest comparisons. There are many of them, and textbooks often present them as a menu. That invites the worst habit in applied statistics: running several tests and reporting the one that gives the smallest p-value. The better way is to let the design of the study and the question being asked choose the test, before the data are seen. By the end of the lesson you will have a short decision guide, and a simulation showing what shopping among tests costs.

IntuitionH

Signal, noise, and which noise counts

Every comparison has the same skeleton: a signal (the difference between the groups) divided by the noise against which it must be judged (the standard error of that difference). What changes from design to design is which noise counts.

Consider twelve runners who each run 5 km twice, once in shoe A and once in shoe B. Runners differ enormously: the fastest finish in about 20 minutes, the slowest in nearly 28. The shoe makes a difference of about 12 seconds. If we throw the 24 times into two piles labelled A and B, the 12-second signal is buried under minutes of differences between runners. But each runner wore both shoes, so we can compare each runner with themselves. The differences between runners cancel, and the shoe effect stands out.

shoe Ashoe Bas two separate groups1200130014001500160017005 km time (s)120013001400150016001700shoe Ashoe Bas pairs, one line per runner
The same 24 race times (synthetic data), viewed two ways. As two separate groups (left), the shoes look indistinguishable, because runners differ by several minutes. Joined as pairs (right), almost every runner is a little faster in shoe B: the comparison within each runner removes the differences between runners.

This is why the first question is always how the groups were formed. In independent groups, different units (plots, people, patients) are in each group, and the noise includes all the variation between units. In a paired (or matched) design, each unit appears in both conditions, or units are deliberately matched in pairs, and the noise is only the variation within pairs. The second question is what kind of outcome was measured: a number (yield, time, length of stay) or a category (returned or not, type of transport).

DefinitionsH

Definitions

Definition— Welch's two-sample t-test

For independent samples of sizes n1,n2n_1, n_2 with means xˉ1,xˉ2\bar x_1, \bar x_2 and standard deviations s1,s2s_1, s_2, Welch's statistic is t=xˉ2−xˉ1s12/n1+s22/n2,t = \frac{\bar x_2 - \bar x_1}{\sqrt{s_1^2/n_1 + s_2^2/n_2}}, compared with a tt distribution whose degrees of freedom are estimated by the Welch–Satterthwaite formula ν=(s12/n1+s22/n2)2(s12/n1)2n1−1+(s22/n2)2n2−1.\nu = \frac{\left(s_1^2/n_1 + s_2^2/n_2\right)^2}{\dfrac{(s_1^2/n_1)^2}{n_1 - 1} + \dfrac{(s_2^2/n_2)^2}{n_2 - 1}}. Student's pooled test instead combines the two variances into one, which is valid only if the populations have equal variances.

Definition— paired t-test

For pairs (xi,yi)(x_i, y_i), compute the differences di=yi−xid_i = y_i - x_i and apply the one-sample tt-test to them: t=dˉ/(sd/n)t = \bar d / (s_d/\sqrt n) with n−1n - 1 degrees of freedom.

Definition— chi-squared test of independence

For a table of counts with rr rows and cc columns, the expected count in each cell under the null hypothesis of no association is (row total × column total) / grand total. Pearson's chi-squared statistic is X2=∑cells(O−E)2E,X^2 = \sum_{\text{cells}} \frac{(O - E)^2}{E}, compared with a chi-squared distribution with (r−1)(c−1)(r-1)(c-1) degrees of freedom. For a 2×2 table it equals the square of the two-proportion zz statistic.

Definition— Mann–Whitney test

The Mann–Whitney (or Wilcoxon rank-sum) test pools the two samples, ranks all the values, and asks whether one group's ranks are systematically higher. It tests whether P(X>Y)P(X > Y), the chance that a random member of one group exceeds a random member of the other, equals 12\tfrac12.

Definition— permutation test

A permutation test computes a statistic (such as the difference in means), then recomputes it after randomly reshuffling the group labels many times. The p-value is the share of reshuffles giving a statistic at least as extreme as the observed one. It is valid when, under the null hypothesis, the labels are exchangeable: any assignment of labels to the observed values was equally likely, as it is in a randomised experiment where the treatment truly does nothing.

ComputationC

Two independent groups: Welch's t-test

An agricultural research station compares a local sorghum variety on 12 plots with an improved variety on 15 plots (synthetic data). We compute Welch's test by hand and with SciPy, then ask how Student's pooled test behaves when the two groups have different spreads.

python
import numpy as np
from scipy import stats

# Synthetic sorghum yields (t/ha) from plots of two varieties at a research station.
rng = np.random.default_rng(51)
local = np.round(rng.normal(2.9, 0.35, 12), 2)
improved = np.round(rng.normal(3.3, 0.6, 15), 2)

def welch(a, b):
    """Welch's t statistic and its Welch-Satterthwaite degrees of freedom."""
    va, vb = a.var(ddof=1) / a.size, b.var(ddof=1) / b.size
    t = (b.mean() - a.mean()) / np.sqrt(va + vb)
    df = (va + vb) ** 2 / (va**2 / (a.size - 1) + vb**2 / (b.size - 1))
    return t, df

t, df = welch(local, improved)
p = 2 * stats.t.sf(abs(t), df)
print(f"local mean {local.mean():.3f} (sd {local.std(ddof=1):.3f}), improved mean {improved.mean():.3f} (sd {improved.std(ddof=1):.3f})")
print(f"by hand: t = {t:.3f}, df = {df:.1f}, p = {p:.4f}")
res = stats.ttest_ind(improved, local, equal_var=False)
ci = res.confidence_interval(0.95)
print(f"SciPy:   t = {res.statistic:.3f}, p = {res.pvalue:.4f}, 95% CI for the difference ({ci.low:.3f}, {ci.high:.3f}) t/ha")

# Student's pooled t-test assumes equal variances. How often does each test reject a TRUE null
# when the smaller group is the more variable one?
reps = 20_000
small = rng.normal(0.0, 3.0, size=(reps, 10))      # n = 10, sd 3
large = rng.normal(0.0, 1.0, size=(reps, 40))      # n = 40, sd 1, same mean
p_student = stats.ttest_ind(small, large, axis=1, equal_var=True).pvalue
p_welch = stats.ttest_ind(small, large, axis=1, equal_var=False).pvalue
print(f"\nfalse-positive rate at alpha = 0.05: Student {np.mean(p_student < 0.05):.3f}, Welch {np.mean(p_welch < 0.05):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The improved variety yielded 0.49 t/ha more on average, with a 95% confidence interval from 0.07 to 0.91 t/ha and p=0.024p = 0.024. The interval is the main result: the improvement is probably real, but it could be anything from negligible to almost a tonne per hectare, and a seed programme would want more trials before relying on the upper end.

The second part shows why Welch's test is the better default. When the smaller group is the more variable one, Student's test rejects a true null hypothesis 25% of the time instead of 5%, because pooling the variances underestimates the noise in the small, variable group. Welch's test holds its level at 5.1%. When the variances happen to be equal, Welch's test loses almost nothing. That is why R's t.test uses Welch by default, and why statisticians recommend it routinely (Delacre, Lakens and Leys, 2017).

ComputationC

Paired designs

Here are the runners from the figure. Each ran once in each shoe, in random order, so the natural analysis works with each runner's difference.

python
import numpy as np
from scipy import stats

# Synthetic 5 km times (seconds) for 12 runners, each running once in shoe A and once in shoe B
# (in random order, on similar days). Runners differ a lot; the shoe effect is small.
rng = np.random.default_rng(8)
ability = rng.normal(1500, 120, 12)                  # each runner's typical time
shoe_a = np.round(ability + rng.normal(0, 8, 12))
shoe_b = np.round(ability - 10 + rng.normal(0, 8, 12))
diff = shoe_b - shoe_a
print("runner   A      B    B - A")
for i in range(12):
    print(f"{i + 1:4} {shoe_a[i]:6.0f} {shoe_b[i]:6.0f} {diff[i]:+6.0f}")
print(f"mean difference {diff.mean():.1f} s, sd of differences {diff.std(ddof=1):.1f} s")

paired = stats.ttest_rel(shoe_b, shoe_a)
unpaired = stats.ttest_ind(shoe_b, shoe_a, equal_var=False)
signed_rank = stats.wilcoxon(shoe_b, shoe_a)
print(f"\npaired t-test:            t = {paired.statistic:6.2f}, p = {paired.pvalue:.4f}")
print(f"Welch test, ignoring pairs: t = {unpaired.statistic:6.2f}, p = {unpaired.pvalue:.4f}")
print(f"Wilcoxon signed-rank test:  p = {signed_rank.pvalue:.4f}")
print(f"correlation between a runner's two times: {np.corrcoef(shoe_a, shoe_b)[0, 1]:.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Analysed as pairs, the evidence is strong: a mean improvement of 12.2 seconds with p=0.0009p = 0.0009. Analysed as if the two columns were independent groups, the same numbers give p=0.84p = 0.84, nothing at all. The Wilcoxon signed-rank test, a rank-based version of the paired test that does not assume the differences are normal, agrees with the paired tt-test (p=0.0033p = 0.0033).

Pairing is a choice made when designing a study, and it is one of the most powerful choices available: measuring the same patients before and after, the same plots in two seasons, or twins in a twin study. But the analysis must follow the design. Treating paired data as independent usually wastes the pairing, as here, and if the two measurements in a pair are negatively correlated it can even make results look more significant than they are. Treating independent data as paired is simply meaningless: there is no natural way to pair them.

ComputationC

Counts and proportions

When the outcome is a category, the data form a table of counts. A district health office ran a randomised trial: 500 families received an SMS reminder before their child's second vaccine dose was due, and 500 did not.

python
import numpy as np
from scipy import stats

# A randomised trial of SMS reminders: did parents bring their child back for the second dose?
#                    returned  did not return
table = np.array([[412, 88],      # SMS reminder (500 families)
                  [365, 135]])    # no reminder  (500 families)
n1, n2 = table.sum(axis=1)
p1, p2 = table[0, 0] / n1, table[1, 0] / n2
print(f"return rates: reminder {p1:.3f}, no reminder {p2:.3f}, difference {p1 - p2:.3f}")

# Two-proportion z-test, using the pooled proportion under H0: p1 = p2.
pooled = table[:, 0].sum() / table.sum()
se0 = np.sqrt(pooled * (1 - pooled) * (1 / n1 + 1 / n2))
z = (p1 - p2) / se0
print(f"z = {z:.3f}, two-sided p = {2 * stats.norm.sf(abs(z)):.5f}")

# The chi-squared test of the same table compares observed counts with expected counts.
chi2, p, df, expected = stats.chi2_contingency(table, correction=False)
print(f"chi-squared = {chi2:.3f} (= z squared, {z**2:.3f}), df = {df}, p = {p:.5f}")
print("expected counts if reminders made no difference:\n", np.round(expected, 1))

# An effect size with its interval: the difference in proportions (unpooled standard error).
se = np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)
print(f"difference {p1 - p2:.3f}, 95% CI ({p1 - p2 - 1.96 * se:.3f}, {p1 - p2 + 1.96 * se:.3f}); "
      f"risk ratio {p1 / p2:.3f}")

# With small counts the chi-squared approximation is poor; Fisher's exact test is exact.
pilot = np.array([[9, 3], [3, 7]])
print("\npilot study expected counts:\n", np.round(stats.chi2_contingency(pilot)[3], 2))
print(f"Fisher's exact test p = {stats.fisher_exact(pilot).pvalue:.4f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

With reminders, 82.4% of children came back, against 73.0% without, a difference of 9.4 percentage points. The two-proportion zz-test and the chi-squared test are the same test in different clothes (X2=z2=12.75X^2 = z^2 = 12.75, p=0.0004p = 0.0004), and both compare the observed counts with the counts expected if reminders made no difference (388.5 returns in each arm).

The report a health office can act on is the effect size: reminders raised the return rate by 9.4 points (95% CI 4.3 to 14.5), a risk ratio of 1.13. Translated into children, that is roughly 9 more returns for every 100 families sent a message, which, set against the cost of the messages, is what decides whether to adopt them. For rare outcomes, epidemiologists also report odds ratios, which logistic regression estimates (in Generalised linear models).

The chi-squared test relies on a large-sample approximation. The usual rule asks for expected counts of at least 5 in every cell; the pilot study's smallest expected count is 4.55, so we use Fisher's exact test instead, which computes the p-value exactly from all tables with the same margins (p=0.084p = 0.084). The chi-squared test also extends to bigger tables, such as preferred transport mode (bus, minibus, motorcycle, walking) by city, with (r−1)(c−1)(r-1)(c-1) degrees of freedom. For paired yes/no data, such as the same people asked before and after a campaign, the right tool is McNemar's test, which uses only the people whose answer changed.

ComputationC

Without the normal assumption: ranks and permutations

tt-tests lean on the central limit theorem, and with small, skewed samples that support can be weak, as Sampling distributions and the CLT showed. Two families of tests avoid the normal assumption.

A permutation test builds the null distribution from the data themselves. If the ward made no difference to length of stay, the labels "ward A" and "ward B" are arbitrary: any shuffling of the 33 stays into groups of 15 and 18 was as likely as the one observed. So we shuffle thousands of times, recompute the difference in means each time, and see where the real difference falls. The idea is Ronald Fisher's, from the 1930s, but it became practical only with computers. Here it is, implemented from scratch, alongside Welch's test and the Mann–Whitney test on the same data, followed by a simulation of what happens when an analyst runs several tests and reports the best.

python
import numpy as np
from scipy import stats

# Synthetic lengths of hospital stay (days) on two wards: skewed, with a few long stays.
rng = np.random.default_rng(34)
ward_a = np.round(rng.lognormal(1.3, 0.6, 15), 1)
ward_b = np.round(rng.lognormal(1.75, 0.6, 18), 1)
print("ward A:", np.sort(ward_a))
print("ward B:", np.sort(ward_b))
observed = ward_b.mean() - ward_a.mean()
print(f"difference in means (B - A): {observed:.2f} days")

def permutation_test(a, b, n_perm, rng):
    """Two-sided permutation test for a difference in means."""
    pooled = np.concatenate([a, b])
    obs = abs(b.mean() - a.mean())
    count = 0
    for _ in range(n_perm):
        shuffled = rng.permutation(pooled)            # relabel at random, as H0 allows
        diff = shuffled[a.size:].mean() - shuffled[:a.size].mean()
        count += abs(diff) >= obs
    return (count + 1) / (n_perm + 1)                 # count the observed labelling too

print(f"permutation test (20,000 relabellings): p = {permutation_test(ward_a, ward_b, 20_000, rng):.4f}")
print(f"Welch t-test:                            p = {stats.ttest_ind(ward_b, ward_a, equal_var=False).pvalue:.4f}")
print(f"Mann-Whitney U test:                     p = {stats.mannwhitneyu(ward_b, ward_a).pvalue:.4f}")

# The forking path: run four tests on data with NO real difference, and keep the smallest p.
reps = 4_000
a = rng.lognormal(1.5, 0.8, size=(reps, 15))
b = rng.lognormal(1.5, 0.8, size=(reps, 18))
ps = np.vstack([
    stats.ttest_ind(a, b, axis=1, equal_var=False).pvalue,
    stats.ttest_ind(a, b, axis=1, equal_var=True).pvalue,
    stats.mannwhitneyu(a, b, axis=1).pvalue,
    stats.ttest_ind(np.log(a), np.log(b), axis=1).pvalue,
])
print(f"\nfalse-positive rate of each test alone: {np.round(np.mean(ps < 0.05, axis=1), 3)}")
print(f"false-positive rate of 'whichever test gives the smallest p': {np.mean(ps.min(axis=0) < 0.05):.3f}")
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

The three tests disagree, and the disagreement is instructive. Ward A has one patient who stayed 17.2 days. That single long stay raises ward A's mean and inflates its variance, so Welch's test gives p=0.052p = 0.052. The permutation test, which compares the same means but builds its null distribution by reshuffling rather than from the tt distribution, gives 0.040. The Mann–Whitney test works with ranks, so it does not care whether the long stay was 17.2 days or 12: it sees that ward B's stays mostly rank above ward A's, and gives 0.005. They answer different questions. Welch's and the permutation test ask about the mean length of stay, which is what drives bed-days and costs; Mann–Whitney asks whether a typical patient on ward B stays longer than a typical patient on ward A.

The second part shows why the choice must come first. On data with no real difference, each test alone gives false positives at about its nominal rate (3.8% to 5.0%). An analyst who runs all four and reports the smallest p-value gets a false-positive rate of 7.3%. The tests are highly correlated, so the inflation is modest here; with more tests, more outcomes and more subgroups the forking paths multiply quickly.

ExplanationH

Choose the test by the design

The design and the question, not the p-value, should pick the test. The guide below covers the cases in this lesson; when the outcome is a measurement and the samples are small and strongly skewed, the bracketed alternatives are worth planning for in advance.

What is the outcome?a measurementa category (yes/no, type)independent groupspaired or matchedindependent groupspaired or matchedWelch t-test(Mann–Whitney orpermutation test ifskewed or small)paired t-test onthe differences(Wilcoxon signed-rankif skewed)chi-squared ortwo-proportion z-test(Fisher's exact testif counts are small)McNemar's teston the pairs thatdisagree
A decision guide for comparing two groups. First ask what kind of outcome was measured, then how the groups were formed. The test in each green box is the usual default; the alternative in brackets is chosen in advance when its condition is expected, not after looking at p-values.

Three further points make the guide work in practice.

  • Decide what you want to compare. Means (for totals and costs), medians or typical values (for skewed outcomes such as income or waiting time), or proportions. This, more than distributional assumptions, decides between Welch and Mann–Whitney.
  • Write it down before looking. The test, the outcome, the comparison and any transformations go in the analysis plan. If the data later turn out to violate an assumption badly, say so and report both analyses, rather than quietly switching.
  • More than two groups is a different problem. Comparing four fertilisers by running six pairwise tt-tests inflates false positives; the methods for that, analysis of variance and its follow-up comparisons, are in Experimental design and ANOVA. And every test in this lesson is a special case of a regression model, which is how they are usually extended to adjust for other variables (Linear regression onwards).
ExplanationH

Comparing groups in professional practice

Clinical trials compare treatments under a protocol written in advance, report the primary comparison with its confidence interval, and follow reporting guidelines such as CONSORT. Online experiments in technology companies compare conversion rates or revenue between randomised groups of users, at enormous scale; the Data Science elective's experiments in products covers their practical pitfalls. Agricultural field trials, where many of these methods were born, compare varieties on plots arranged in blocks so that plots in similar soil are compared with each other, the field version of pairing.

Effect sizes for comparisons. Report the difference in means (in real units) with its interval; for proportions, the risk difference and the risk ratio (and the number of people who need to be treated for one to benefit, 1/risk difference1/\text{risk difference}, about 11 families per extra return in the SMS trial). Standardised effect sizes such as Cohen's dd help when combining studies measured on different scales.

Software. SciPy provides ttest_ind (pass equal_var=False for Welch), ttest_rel, wilcoxon, mannwhitneyu, chi2_contingency, fisher_exact and a general permutation_test. In R, shown for comparison and not run:

r
# R, not run here
t.test(yield ~ variety, data = plots)          # Welch by default
t.test(shoe_b, shoe_a, paired = TRUE)
wilcox.test(stay ~ ward, data = stays)          # Mann-Whitney
chisq.test(matrix(c(412, 365, 88, 135), 2), correct = FALSE)
fisher.test(matrix(c(9, 3, 3, 7), 2))

History. Karl Pearson introduced the chi-squared test in 1900. Student's tt (1908) was extended to unequal variances by Bernard Welch in 1947, the same year Henry Mann and Donald Whitney published their rank test (Frank Wilcoxon had proposed an equivalent test in 1945). Fisher described the logic of permutation tests in the 1930s, when the reshuffling had to be done by hand for tiny samples; today a laptop does tens of thousands of reshuffles in about a second.

Common misconceptionsH

Common misconceptions

Misconception

"Check normality and equal variances first, then choose the test." Two-stage procedures distort error rates, and preliminary tests have little power in small samples. Choose by design and question in advance; use Welch's test as the default for independent means.

Misconception

"The Mann–Whitney test compares medians." It tests whether values in one group tend to exceed those in the other, P(X>Y)=12P(X > Y) = \tfrac12. Only if the two distributions have the same shape and differ by a shift is that the same as comparing medians. Two groups with equal medians but different shapes can give a significant Mann–Whitney test.

Misconception

"If two confidence intervals overlap, the difference is not significant." Two 95% intervals can overlap noticeably while the difference between the groups is significant at 5%, because the standard error of a difference is smaller than the sum of the two standard errors. Compute the interval for the difference itself.

Misconception

"Treatment worked in women (p=0.01p = 0.01) but not in men (p=0.32p = 0.32), so it works differently by sex." Suppose the effect is 2.0 (standard error 0.8) in one group and 0.8 (standard error 0.8) in the other. The difference between the two effects is 1.2 with standard error 1.13, giving p=0.29p = 0.29. The difference between "significant" and "not significant" is not itself significant (Gelman and Stern, 2006); to claim the groups differ, test the difference.

Misconception

"Analysing paired data as independent just loses a little power." It uses the wrong standard error. With positively correlated pairs it usually throws away most of the information (in the shoe example, p=0.84p = 0.84 instead of 0.00090.0009); with negatively correlated pairs it can overstate the evidence.

Exercises

Exercises

1. Comparing groups

Warm-upHuman

A clinic in Kumasi measures the systolic blood pressure of 30 patients, then again after each has followed a three-month exercise programme. Which analysis matches the design?

Choose one

2. Comparing groups

CoreHuman

A survey of 200 adults in Tanzania records where they live and whether they use mobile money.

uses mobile money does not total
urban 102 18 120
rural 48 32 80
total 150 50 200

For the chi-squared test of independence, what is the expected number of urban adults who use mobile money?

3. Comparing groups

CoreHuman

An exact permutation test compares two groups of 5 measurements each by considering every way of splitting the 10 pooled values into a group of 5 labelled A and a group of 5 labelled B. How many splits are there?

4. Comparing groups

CoreHuman

A Mann–Whitney test comparing waiting times at two clinics gives p=0.004p = 0.004. Which conclusion does the test itself support?

Choose one

5. Comparing groups

CoreHuman

An analyst comparing two groups runs Welch's test (p=0.08p = 0.08), Student's test (p=0.07p = 0.07), a Mann–Whitney test (p=0.03p = 0.03) and a tt-test on logarithms (p=0.05p = 0.05). She reports only the Mann–Whitney result as "significant at the 5% level". What is the main problem?

Choose one

6. Comparing groups

CoreComputational

Write permutation_p_value(a, b, n_perm, seed), a two-sided permutation test for a difference in means:

  1. Pool a and b. Compute the observed statistic abs(mean(b) - mean(a)).
  2. With rng = np.random.default_rng(seed), repeat n_perm times: shuffle the pooled values (rng.permutation), call the first len(a) of them group A and the rest group B, and compute the same statistic.
  3. Count the reshuffles whose statistic is at least the observed one (compare with >= observed - 1e-9, which protects against rounding), and return (count + 1) / (n_perm + 1), which counts the observed labelling as one of the possibilities.
Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

7. Comparing groups

StretchComputational

Write welch_test(a, b) returning a tuple of three floats (t, df, p): Welch's statistic for mean(b) - mean(a), the Welch–Satterthwaite degrees of freedom, and the two-sided p-value. Use the sample variances (with ddof=1) and scipy.stats.t.sf for the tail probability, but not ttest_ind itself. The tests compare your results with SciPy's.

Runs entirely in your browser, in an isolated worker. Nothing is sent to a server.

Further reading and resources