📐 Concept diagram

Phase 10: Probability Theory

Subject 10-03: Independence

Prerequisites: 10-01 (Probability Foundations), 10-02 (Conditional Probability)

Learning Objectives

1. State the definition of independence for two events and verify independence using P(A ∩ B) = P(A) P(B)
2. Distinguish pairwise independence from mutual independence with counterexamples
3. Apply independence to calculate probabilities in repeated trials and product spaces
4. Define and apply conditional independence P(A ∩ B | C) = P(A | C) P(B | C)
5. Recognize common fallacies: disjointness vs. independence, transitivity of independence

Core Content

1. Definition: Two Events

Events A and B are independent if:

$P(A ∩ B) = P(A) · P(B)
$

Equivalent formulations (when probabilities are positive):

• P(A|B) = P(A) — knowing B occurred doesn't change the probability of A
• P(B|A) = P(B) — knowing A occurred doesn't change the probability of B
• P(A|B) = P(A|Bᶜ) — A is equally likely regardless of B

Proof of equivalence: If P(B) > 0, then P(A ∩ B) = P(A) P(B) ⇔ P(A|B) = P(A ∩ B)/P(B) = P(A) P(B)/P(B) = P(A). The other formulations follow similarly. ∎

Intuition: Independence means that knowing whether one event occurred gives zero information about the other. They are "probabilistically unrelated."

Critical distinction: Independence ≠ disjointness.

• If A and B are disjoint (A ∩ B = ∅), then P(A ∩ B) = 0, so independence would require P(A)P(B) = 0, meaning at least one has probability zero. Disjoint events are (usually) extremely DEPENDENT: if A occurs, B cannot occur.
• If A and B are independent, then P(A ∩ B) = P(A)P(B) > 0 (assuming both positive), so they CAN occur together.

Key principle: Disjoint events carry maximal negative information about each other. Independent events carry zero information.

2. Independence of Complements

Theorem: If A and B are independent, then so are: - A and Bᶜ - Aᶜ and B - Aᶜ and Bᶜ

Proof for A and Bᶜ: P(A ∩ Bᶜ) = P(A) − P(A ∩ B) = P(A) − P(A)P(B) = P(A)(1 − P(B)) = P(A)P(Bᶜ). ∎

Similarly, all four pairs are independent. Once you know one pair, you know all complement pairs.

3. Mutual Independence (Three or More Events)

For events A₁, A₂, A₃, mutual independence requires ALL of:

$P(A₁ ∩ A₂) = P(A₁) P(A₂)
P(A₁ ∩ A₃) = P(A₁) P(A₃)
P(A₂ ∩ A₃) = P(A₂) P(A₃)
P(A₁ ∩ A₂ ∩ A₃) = P(A₁) P(A₂) P(A₃)
$

All four equations must hold. The first three alone define pairwise independence.

General definition (n events): Events A₁, ..., Aₙ are mutually independent if for every subset {i₁, ..., iₖ} ⊆ {1, ..., n}:

$P(A_{i₁} ∩ ... ∩ A_{iₖ}) = P(A_{i₁}) · ... · P(A_{iₖ})
$

This requires 2ⁿ − n − 1 conditions to be checked (all subsets of size ≥ 2).

4. Pairwise vs. Mutual: Famous Counterexample (Bernstein)

Two fair coins are tossed. Define events:

• A: first coin is heads = {HH, HT}
• B: second coin is heads = {HH, TH}
• C: the two coins match = {HH, TT}

Check pairwise: - P(A) = 1/2, P(B) = 1/2, P(A ∩ B) = P(HH) = 1/4 = (1/2)(1/2) ✓ - P(A ∩ C) = P(HH) = 1/4 = (1/2)(1/2) ✓ - P(B ∩ C) = P(HH) = 1/4 = (1/2)(1/2) ✓

All pairwise independent!

But mutual independence fails: - P(A ∩ B ∩ C) = P(HH) = 1/4 - P(A) P(B) P(C) = (1/2)(1/2)(1/2) = 1/8 - 1/4 ≠ 1/8 ✗

So A, B, C are pairwise independent but NOT mutually independent. Knowing any two of them tells you the third deterministically! If you know the first coin is heads AND the coins match, the second MUST be heads.

Lesson: Checking only pairwise independence is insufficient. You need the intersection of ALL subsets to factor.

5. Conditional Independence

Events A and B are conditionally independent given C (with P(C) > 0) if:

$P(A ∩ B | C) = P(A | C) · P(B | C)
$

Equivalently: P(A | B ∩ C) = P(A | C) (when P(B ∩ C) > 0).

Important: Conditional independence does NOT imply unconditional independence, and vice versa.

Example where conditional independence holds but unconditional does not:

Suppose a student's exam performance (A = pass) and whether they studied (B) are related. Let C = "student is well-rested." It's possible that among well-rested students, studying has no additional benefit (A independent of B given C), but overall, both studying and rest correlate with passing, so A and B are unconditionally dependent.

Example where unconditional independence holds but conditional does not (Simpson's paradox territory):

Two treatments might appear equally effective overall (independent of treatment), but for each severity subgroup, one treatment is better — the composition of severity differs between treatment groups.

6. Applications of Independence

Repeated independent trials: If an experiment with P(success) = p is repeated independently n times:

P(exactly k successes) = C(n,k) · pᵏ · (1-p)^{n-k}

Reliability of systems: - Series system with independent components (all must work): P(system works) = P(A₁) P(A₂) ... P(Aₙ) - Parallel system with independent components (at least one must work): P(system works) = 1 − P(all fail) = 1 − (1-P(A₁))(1-P(A₂))...(1-P(Aₙ))

Products of probability spaces: When n independent experiments are combined, the joint probability space is the Cartesian product Ω = Ω₁ × Ω₂ × ... × Ωₙ with probability measure P(A₁ × A₂ × ... × Aₙ) = P₁(A₁) P₂(A₂) ... Pₙ(Aₙ).

Key Terms

• Equivalent formulations

Worked Examples

Example 1: Checking Independence from Probabilities

Given P(A) = 0.5, P(B) = 0.4, P(A ∪ B) = 0.7. Are A and B independent?

Solution:

P(A ∩ B) = P(A) + P(B) − P(A ∪ B) = 0.5 + 0.4 − 0.7 = 0.2. P(A)P(B) = (0.5)(0.4) = 0.2.

Since P(A ∩ B) = P(A)P(B) = 0.2, A and B are independent. ✓

Example 2: Pairwise Does Not Imply Mutual (card draw version)

Draw one card from a standard 52-card deck. Let: A = {card is a heart} B = {card is a face card (J, Q, K)} C = {card is a king}

Show that A and B are independent, A and C are independent, but A, B, C are not mutually independent.

Solution:

P(A) = 13/52 = 1/4 P(B) = 12/52 = 3/13 P(C) = 4/52 = 1/13

P(A ∩ B) = P(heart AND face) = 3/52. P(A)P(B) = (1/4)(3/13) = 3/52. ✓ Independent. P(A ∩ C) = P(king of hearts) = 1/52. P(A)P(C) = (1/4)(1/13) = 1/52. ✓ Independent.

Now check mutual independence: P(A ∩ B ∩ C) = P(king of hearts) = 1/52. P(A)P(B)P(C) = (1/4)(3/13)(1/13) = 3/676.

1/52 = 13/676 ≠ 3/676. Not mutually independent.

Why? Because A ∩ B ∩ C = {king of hearts} has probability 1/52, but the product of individual probabilities is much smaller. Knowing the heart AND the face card status over-constrains the king event.

Example 3: System Reliability

A system has three independent components. P(component works) = 0.9, 0.8, 0.95 respectively.

(a) If components are in series (all must work), what is P(system works)? (b) If in parallel (at least one must work), what is P(system works)? (c) Design a system that works if component 1 works AND at least one of components 2 or 3 works.

Solution:

(a) Series: P(works) = (0.9)(0.8)(0.95) = 0.684.

(b) Parallel: P(works) = 1 − P(all fail) = 1 − (0.1)(0.2)(0.05) = 1 − 0.001 = 0.999.

(c) P(works) = P(C₁ works) · P(C₂ works OR C₃ works) = 0.9 · [1 − P(C₂ fails)P(C₃ fails)] = 0.9 · [1 − (0.2)(0.05)] = 0.9 · 0.99 = 0.891.

Quiz

Q1: Two events A and B with P(A) > 0 and P(B) > 0 are independent if:

A) They are mutually exclusive (disjoint) B) P(A ∩ B) = P(A) × P(B) C) P(A ∪ B) = P(A) + P(B) D) P(A|B) = P(B|A)

Correct: B)

• If you chose B: Correct! Independence is defined by P(A ∩ B) = P(A)P(B). Equivalently, P(A|B) = P(A) when P(B) > 0.
• If you chose A: Disjoint events are usually extremely DEPENDENT — if A occurs, B cannot occur. For independent events with positive probability, P(A ∩ B) = P(A)P(B) > 0, meaning they CAN occur together.
• If you chose C: This holds for disjoint events, not independent ones. For independent events, P(A ∪ B) = P(A) + P(B) − P(A)P(B).
• If you chose D: This is symmetry of conditional probability, which is not implied by independence (and generally false).

Q2: In Bernstein's famous counterexample with two fair coins, events A={first heads}, B={second heads}, C={coins match} are:

A) Mutually independent B) Pairwise independent but not mutually independent C) Dependent in all pairs D) Conditionally independent only

Correct: B)

• If you chose B: Correct! P(A∩B)=P(A∩C)=P(B∩C)=1/4 = (1/2)(1/2), so all pairs are independent. But P(A∩B∩C)=1/4 ≠ P(A)P(B)P(C)=1/8, so mutual independence fails. Knowing any two determines the third.
• If you chose A: The triple intersection doesn't factor, so mutual independence fails.
• If you chose C: Each pair passes the independence test (1/4 = 1/2 × 1/2).
• If you chose D: This is not about conditional independence; the events fail unconditional mutual independence.

Q3: If A and B are independent, which pair is guaranteed to also be independent?

A) A and A B) A and B ∪ A C) Aᶜ and Bᶜ D) A ∩ B and A ∪ B

Correct: C)

• If you chose C: Correct! Independence is preserved under complement: P(Aᶜ ∩ Bᶜ) = 1 − P(A) − P(B) + P(A∩B) = 1 − P(A) − P(B) + P(A)P(B) = (1−P(A))(1−P(B)) = P(Aᶜ)P(Bᶜ).
• If you chose A: A is independent of itself only if P(A) = 0 or 1, since P(A∩A) = P(A) = P(A)² only when P(A) ∈ {0,1}.
• If you chose B: These events overlap heavily and are typically dependent.
• If you chose D: These are generally dependent since they share probability mass.

Q4: For three events to be mutually independent, how many equations must be checked?

A) 3 B) 4 C) 6 D) 1

Correct: B)

• If you chose B: Correct! We need C(3,2)=3 pairwise equations plus C(3,3)=1 triple intersection equation = 4 total.
• If you chose A: Three equations covers only pairwise independence, which is insufficient.
• If you chose C: Six would be the total number of non-empty subsets, but singleton conditions aren't needed.
• If you chose D: One equation (even the triple) is insufficient without the pairwise checks.

Q5: Conditional independence P(A ∩ B | C) = P(A | C) P(B | C) means:

A) A and B are independent overall B) A and B are independent within the subpopulation where C occurs C) A and B are independent when C does NOT occur D) A, B, and C are mutually independent

Correct: B)

• If you chose B: Correct! Conditional independence restricts the independence relationship to the world where C is true.
• If you chose A: Conditional independence does NOT imply unconditional independence (and vice versa — Simpson's paradox).
• If you chose C: This describes independence given Cᶜ, which is a separate condition.
• If you chose D: Conditional independence of A and B given C does not require independence involving C.

Q6: Three independent components each have reliability 0.9 connected in parallel. The system reliability is:

A) 0.9³ = 0.729 B) 1 − 0.1³ = 0.999 C) 0.9 D) 1 − 0.9³ = 0.271

Correct: B)

• If you chose B: Correct! A parallel system fails only if ALL components fail. P(all fail) = 0.1³ = 0.001, so reliability = 1 − 0.001 = 0.999.
• If you chose A: This is the reliability for a SERIES system (all must work).
• If you chose C: This is the reliability of a single component.
• If you chose D: This is the probability that at least one component fails: 1 − 0.729.

Q7: Independence is a transitive relation: if A is independent of B and B is independent of C, then A is independent of C.

A) True B) False

Correct: B)

• If you chose B: Correct! Independence is NOT transitive. Counterexample: Let A and B be independent fair coin flips, and set C = A. Then A⊥B and B⊥C, but A and C are identical (perfectly dependent).
• If you chose A: This is a common misconception. Independence does not chain. Counterexamples abound.

Practice Problems

1. If A and B are independent with P(A) = 0.3, P(B) = 0.6, find P(A ∩ B), P(A ∪ B), P(A|B), and P(A ∩ Bᶜ).

2. Show that if A and B are mutually exclusive events with P(A) > 0 and P(B) > 0, then they cannot be independent.

3. Prove: If A, B, C are mutually independent, then A and B ∪ C are independent.

4. Toss a fair coin 4 times. Let A = {first toss is H}, B = {exactly two heads total}, C = {last two tosses are both H}. Are A and B independent? A and C? B and C?

5. A system has 5 independent components, each with reliability 0.95. Find the reliability if: (a) all in series, (b) all in parallel, (c) it's a "3-out-of-5" system (works if at least 3 work).

6. Give an example of three events that are NOT pairwise independent but ARE mutually independent. (Hint: This is impossible — mutual independence implies pairwise. So the exercise is to explain why.)

7. If A and B are conditionally independent given C, and also given Cᶜ, does it follow that A and B are unconditionally independent? Provide a counterexample or proof.

Summary

• A and B are independent iff P(A ∩ B) = P(A)P(B), which means knowing one gives no information about the other
• Disjointness and independence are fundamentally different: disjoint events carry maximal information about each other; independent events carry none
• Mutual independence requires ALL subset intersections to factor — pairwise independence alone is insufficient (Bernstein's coin example)
• Conditional independence P(A ∩ B | C) = P(A | C)P(B | C) is distinct from unconditional independence — neither implies the other
• Independence enables product rules for repeated trials and simplifies reliability calculations for systems of components

Pitfalls

• Confusing independence with disjointness. Disjoint events (A ∩ B = ∅) carry maximal information about each other — if one occurs, the other cannot. Disjoint events with positive probability are always dependent. Independent events can and typically do co-occur.
• Assuming pairwise independence implies mutual independence. Bernstein's coin example (two fair coins: A = first heads, B = second heads, C = coins match) shows three events that are pairwise independent but not mutually independent. Always check all subset intersections.
• Thinking independence is transitive. If A⊥B and B⊥C, it does NOT follow that A⊥C. Counterexample: let A and B be independent fair coin flips, and set C = A. Then A⊥B and B⊥C, but A and C are perfectly dependent.
• Forgetting to check all subset conditions for mutual independence. For n events, mutual independence requires 2^n - n - 1 equations (all intersections of size ≥ 2). Missing the triple intersection check led to the Bernstein counterexample.
• Assuming conditional independence implies unconditional independence (or vice versa). Simpson's paradox illustrates that A and B can be conditionally independent given C yet unconditionally dependent, or conditionally dependent yet unconditionally independent. Conditioning changes the probability space.

Quiz

1. If P(A) = 0.4, P(B) = 0.5, and A and B are independent, then P(A ∪ B) = ? a) 0.9 b) 0.7 c) 0.2 d) 1.0 Answer: b. P(A ∪ B) = P(A) + P(B) − P(A)P(B) = 0.4 + 0.5 − 0.2 = 0.7.

2. Which is true about disjoint events with positive probability? a) They are always independent b) They are always dependent c) They may be independent or dependent d) They are conditionally independent Answer: b. P(A ∩ B) = 0 but P(A)P(B) > 0, so they cannot be independent.

3. For three events to be mutually independent, how many equations must be checked? a) 3 b) 4 c) 6 d) 8 Answer: b. Three pairwise equations plus one triple intersection equation: C(3,2) + C(3,3) = 3 + 1 = 4.

4. In Bernstein's example (two fair coins, A=first heads, B=second heads, C=coins match), A and B are: a) Pairwise independent but not mutually independent with C b) Dependent c) Mutually independent with C d) None of the above Answer: a. A and B are pairwise independent (with each other and with C), but the triple {A, B, C} is not mutually independent.

5. Conditional independence of A and B given C means: a) P(A ∩ B) = P(A) P(B) b) P(A ∩ B | C) = P(A | C) P(B | C) c) A and B are independent when C doesn't occur d) P(A | B) = P(A | C) Answer: b. The definition directly involves conditioning everything on C.

6. If A and B are independent, which of the following is also guaranteed to be independent? a) A and A b) Aᶜ and Bᶜ c) A ∪ B and A ∩ B d) All of the above Answer: b. Complement events preserve independence. A with itself is independent only if P(A) is 0 or 1.

7. A component has reliability 0.9. Three such independent components in parallel have combined reliability: a) 0.9³ = 0.729 b) 1 − 0.1³ = 0.999 c) 0.9 d) 0.3 Answer: b. Parallel system fails only if all three fail: P(failure) = 0.1³ = 0.001, so reliability = 0.999.

8. True or False: If A and B are independent and B and C are independent, then A and C are independent. a) True b) False Answer: b. Independence is not transitive. Counterexample: let A and B be independent fair coins, and let C = A. Then A and B independent, B and C independent, but A and C are identical (perfectly dependent).

Next Steps

Continue to 10-04 Discrete Random Variables to learn about PMFs, CDFs, and the fundamental discrete distributions (Bernoulli, binomial, geometric, negative binomial, hypergeometric).