Conditional probability and independence

Lens

MotivationH

You tested positive. Now what?

A screening test for a condition is described as "95% accurate": it detects $95\%$ of people who have the condition, and it gives a correct negative result for $95\%$ of people who do not. About $1$ person in $100$ who take the test has the condition. You take the test, and the result is positive. How likely is it that you have the condition?

Most people answer "about $95\%$ ". The true answer is about $16\%$ .

That is not a trick, and it is not a flaw in the test. It is what happens when we confuse two different questions — "how likely is a positive result, if you have the condition?" and "how likely is the condition, if your result is positive?" — and forget how rare the condition was to begin with. Doctors, juries, journalists and scientists have all made this mistake, sometimes with serious consequences. This lesson gives you the tools to get it right: conditional probability, independence and Bayes' theorem.

IntuitionH

New information shrinks the sample space

Roll two fair dice. The probability that the first die shows a $6$ is $\tfrac16$ . Now suppose a friend peeks and tells you: "the total is $10$ ". Only three of the $36$ equally likely outcomes are left:

(4, 6), \quad (5, 5), \quad (6, 4).

In one of these three the first die shows a $6$ . So, given that the total is $10$ , the probability that the first die is a $6$ has risen from $\tfrac16$ to $\tfrac13$ .

Conditioning on an event $B$ means throwing away every outcome outside $B$ and rescaling, so that the outcomes left over carry all the probability. Here is a second example, from a survey of $400$ secondary-school students in Hanoi about how they travel to school and whether they arrived late at least once last month:

	late	not late	total
cycle	30	170	200
bus	50	150	200
total	80	320	400

Pick a student at random. $P(\text{late}) = \tfrac{80}{400} = 0.2$ . Among the cyclists only, $P(\text{late} \mid \text{cycle}) = \tfrac{30}{200} = 0.15$ ; among the bus riders, $P(\text{late} \mid \text{bus}) = \tfrac{50}{200} = 0.25$ . And among the late students, $P(\text{cycle} \mid \text{late}) = \tfrac{30}{80} = 0.375$ — a quite different number from $0.15$ , because it divides by a different total.

DefinitionsH

Definitions

We work, as in the lesson on probability, with a finite sample space $\Omega$ of outcomes. Each outcome $\omega$ has a probability $p(\omega) \ge 0$ , these add up to $1$ , and the probability of an event $A \subseteq \Omega$ is $P(A) = \sum_{\omega \in A} p(\omega)$ . Equally likely outcomes are the special case $p(\omega) = 1/|\Omega|$ , where $P(A) = |A|/|\Omega|$ .

Definition— conditional probability

For events $A$ and $B$ with $P(B) > 0$ , the conditional probability of $A$ given $B$ is

P(A \mid B) = \frac{P(A \cap B)}{P(B)}.

It is not defined when $P(B) = 0$ .

Definition— independent events

Events $A$ and $B$ are independent if $P(A \cap B) = P(A)\,P(B)$ .

Definition— mutually exclusive events

Events $A$ and $B$ are mutually exclusive if they cannot both happen: $A \cap B = \varnothing$ .

Definition— partition

Events $B_1, \dots, B_n$ partition the sample space if they are pairwise mutually exclusive and together cover it: every outcome lies in exactly one $B_i$ . For example, $B$ and its complement $B^c$ form a partition.

The definition of independence is symmetric in $A$ and $B$ and makes sense even when a probability is $0$ ; Theorem 5 below explains why it captures the idea "knowing $B$ tells us nothing about $A$ ".

Worked examplesH

Worked examples

Example— Two cards without replacement

Draw two cards from a shuffled pack of $52$ , without putting the first back. What is the probability that both are aces? The first is an ace with probability $\tfrac{4}{52}$ . Given that it was, $51$ cards remain, of which $3$ are aces, so the second is an ace with conditional probability $\tfrac{3}{51}$ . Multiplying along the branches of the tree,

P(\text{both aces}) = \frac{4}{52} \times \frac{3}{51} = \frac{12}{2652} = \frac{1}{221}.

The draws are not independent: $P(\text{2nd ace} \mid \text{1st ace}) = \tfrac{3}{51} \ne \tfrac{4}{52} = P(\text{2nd ace})$ .

Example— Independent events from dice

Roll two dice. Let $A$ = "the first die is even" and $B$ = "the total is $7$ ". Then $P(A) = \tfrac{18}{36} = \tfrac12$ , $P(B) = \tfrac{6}{36} = \tfrac16$ , and $A \cap B = \{(2, 5), (4, 3), (6, 1)\}$ , so $P(A \cap B) = \tfrac{3}{36} = \tfrac1{12} = P(A)\,P(B)$ . They are independent: whatever the first die shows, exactly one face of the second die makes the total $7$ .

By contrast, $C$ = "the first die is $6$ " and $D$ = "the total is at least $10$ " are not independent: $P(C \cap D) = \tfrac{3}{36}$ , but $P(C)\,P(D) = \tfrac16 \cdot \tfrac16 = \tfrac{1}{36}$ .

Example— Mutually exclusive is not independent

With one die, "roll a $1$ " and "roll a $6$ " are mutually exclusive. Are they independent? $P(\text{both}) = 0$ , but $P(1)\,P(6) = \tfrac{1}{36} \ne 0$ . No — learning that you rolled a $6$ tells you for certain that you did not roll a $1$ . Mutually exclusive events with positive probabilities are as far from independent as can be.

Example— Pairwise is not enough

Toss two fair coins. Let $A$ = "the first is heads", $B$ = "the second is heads", $C$ = "the two coins show the same face". Each has probability $\tfrac12$ , and each pair is independent: for instance $A \cap C$ = "both heads" has probability $\tfrac14 = \tfrac12 \cdot \tfrac12$ . But $A$ and $B$ together force $C$ : $P(A \cap B \cap C) = \tfrac14 \ne \tfrac18 = P(A)P(B)P(C)$ . For three events to be (mutually) independent we require the product rule for every pair and for all three together.

ProofH

The rules, with proofs

Throughout, $\Omega$ is a finite sample space with probabilities $p(\omega)$ as above.

Theorem— Theorem 1 (basic rules)

For all events $A$ and $B$ : if $A \cap B = \varnothing$ then $P(A \cup B) = P(A) + P(B)$ ; in general

P(A \cup B) = P(A) + P(B) - P(A \cap B), \qquad P(A^c) = 1 - P(A).

Proof

If $A$ and $B$ are disjoint, the sum of $p(\omega)$ over $A \cup B$ splits into the sum over $A$ plus the sum over $B$ . In general, $P(A) + P(B)$ adds $p(\omega)$ twice for each $\omega \in A \cap B$ and once for every other outcome of $A \cup B$ ; subtract $P(A \cap B)$ . Finally $A$ and $A^c$ are disjoint with union $\Omega$ , so $P(A) + P(A^c) = P(\Omega) = 1$ .

∎

Theorem— Theorem 2 (conditioning gives a new probability)

Fix an event $B$ with $P(B) > 0$ . Then $A \mapsto P(A \mid B)$ is again a probability: $0 \le P(A \mid B) \le 1$ , $P(\Omega \mid B) = P(B \mid B) = 1$ , and $P(A_1 \cup A_2 \mid B) = P(A_1 \mid B) + P(A_2 \mid B)$ when $A_1 \cap A_2 = \varnothing$ .

Proof

Since $A \cap B \subseteq B$ , we have $0 \le P(A \cap B) \le P(B)$ ; divide by $P(B)$ . Next, $P(\Omega \cap B) = P(B)$ . If $A_1$ and $A_2$ are disjoint, so are $A_1 \cap B$ and $A_2 \cap B$ , and their union is $(A_1 \cup A_2) \cap B$ ; apply Theorem 1 and divide by $P(B)$ .

∎

In fact $P(\,\cdot \mid B)$ gives each outcome $\omega \in B$ the new probability $p(\omega)/P(B)$ and each outcome outside $B$ probability $0$ : the "throw away and rescale" of the intuition section. So every rule of probability holds for conditional probabilities too — for instance $P(A^c \mid B) = 1 - P(A \mid B)$ .

Theorem— Theorem 3 (multiplication rule)

If $P(B) > 0$ then $P(A \cap B) = P(A \mid B)\,P(B)$ . More generally, if $P(A_1 \cap A_2) > 0$ ,

P(A_1 \cap A_2 \cap A_3) = P(A_1)\,P(A_2 \mid A_1)\,P(A_3 \mid A_1 \cap A_2).

Proof

The first statement is the definition of $P(A \mid B)$ multiplied by $P(B)$ . For the second, write out the right-hand side: $P(A_1) \cdot \dfrac{P(A_1 \cap A_2)}{P(A_1)} \cdot \dfrac{P(A_1 \cap A_2 \cap A_3)}{P(A_1 \cap A_2)}$ ; everything cancels except $P(A_1 \cap A_2 \cap A_3)$ . (Both denominators are positive, since $P(A_1) \ge P(A_1 \cap A_2) > 0$ .)

∎

This is why we multiply along the branches of a tree diagram: each branch carries a conditional probability, given everything on the path before it.

Theorem— Theorem 4 (law of total probability)

If $B_1, \dots, B_n$ partition $\Omega$ and each $P(B_i) > 0$ , then for every event $A$ ,

P(A) = \sum_{i=1}^{n} P(A \mid B_i)\,P(B_i).

In particular, if $0 < P(B) < 1$ : $\;P(A) = P(A \mid B)\,P(B) + P(A \mid B^c)\,P(B^c)$ .

Proof

Every outcome of $A$ lies in exactly one $B_i$ , so $A$ is the union of the pairwise disjoint pieces $A \cap B_1, \dots, A \cap B_n$ , and $P(A) = \sum_i P(A \cap B_i)$ by Theorem 1 (applied repeatedly). By the multiplication rule, $P(A \cap B_i) = P(A \mid B_i)\,P(B_i)$ .

∎

On a tree diagram: to find $P(A)$ , add up the probabilities of all the paths that end in $A$ .

Theorem— Theorem 5 (Bayes' theorem)

If $P(A) > 0$ and $P(B) > 0$ , then

P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B)}.

Combined with Theorem 4, when $0 < P(A) < 1$ :

P(A \mid B) = \frac{P(B \mid A)\,P(A)}{P(B \mid A)\,P(A) + P(B \mid A^c)\,P(A^c)}.

Proof

By the multiplication rule, used twice, $P(A \mid B)\,P(B) = P(A \cap B) = P(B \cap A) = P(B \mid A)\,P(A)$ . Divide by $P(B)$ . For the second form, expand the denominator $P(B)$ by the law of total probability with the partition $A, A^c$ .

∎

Bayes' theorem reverses a conditional probability. It needs one extra ingredient: the base rate (or prior probability) $P(A)$ .

Example— The screening test, solved

Let $D$ = "has the condition" and $T$ = "tests positive". We know $P(D) = 0.01$ , $P(T \mid D) = 0.95$ and $P(T \mid D^c) = 0.05$ . By Bayes' theorem,

P(D \mid T) = \frac{0.95 \times 0.01}{0.95 \times 0.01 + 0.05 \times 0.99} = \frac{0.0095}{0.059} = \frac{19}{118} \approx 0.161.

Natural frequencies make this transparent. Imagine $10\,000$ people. About $100$ have the condition, and $95$ of them test positive. Of the $9\,900$ who do not, $5\%$ — that is $495$ — also test positive. So of the $95 + 495 = 590$ positive results, only $95$ come from people with the condition: $\tfrac{95}{590} \approx 16\%$ . The false positives outnumber the true ones because healthy people are so much more numerous.

The screening test as a tree. Multiply along each path; the two paths ending in a positive result have probabilities $0.0095$ and $0.0495$, so $P(T) = 0.059$ and $P(D \mid T) = 0.0095 / 0.059 \approx 0.16$.

Independence

Theorem— Theorem 6 (independence as irrelevance)

If $P(B) > 0$ , then $A$ and $B$ are independent if and only if $P(A \mid B) = P(A)$ .

Proof

$P(A \mid B) = P(A)$ means $\dfrac{P(A \cap B)}{P(B)} = P(A)$ , which (multiplying by $P(B) > 0$ ) is the same as $P(A \cap B) = P(A)\,P(B)$ .

∎

Theorem— Theorem 7 (complements)

If $A$ and $B$ are independent, so are $A$ and $B^c$ (and hence also $A^c$ and $B$ , and $A^c$ and $B^c$ ).

Proof

$A$ is the disjoint union of $A \cap B$ and $A \cap B^c$ , so

P(A \cap B^c) = P(A) - P(A \cap B) = P(A) - P(A)P(B) = P(A)\big(1 - P(B)\big) = P(A)\,P(B^c),

using independence in the second step and the complement rule in the last. Applying this again with the roles of the events exchanged gives the other two statements.

∎

Theorem— Theorem 8 (exclusive events are dependent)

If $A$ and $B$ are mutually exclusive and $P(A) > 0$ , $P(B) > 0$ , then $A$ and $B$ are not independent.

Proof

$P(A \cap B) = P(\varnothing) = 0$ , while $P(A)\,P(B) > 0$ .

∎

Formal mathematicsF

The rules, checked by Lean

We formalise a finite probability space exactly as in the definitions: a finite type of outcomes and a rational probability for each, adding up to $1$ . Using rational numbers keeps every calculation exact.

FinProbLean 4Formally verified

/-- A **finite probability space** on a finite type of outcomes `Ω`: a probability `p ω ≥ 0`
for each outcome, adding up to `1`. -/
structure FinProb (Ω : Type*) [Fintype Ω] where
  /-- the probability of each single outcome -/
  p : Ω → ℚ
  nonneg : ∀ ω, 0 ≤ p ω
  sum_eq_one : ∑ ω, p ω = 1