What Does Chance Tell Us? The Language of Probability

/ Statistics and Probability

Why This Matters

Flip a fair coin. What’s the probability of heads?

Naive answer: “50%”
Deeper question: What does that 50% actually mean?

It doesn’t mean: “If I flip twice, I’ll get exactly one heads.”
It does mean: “In the long run, about half of flips will be heads.”

Probability quantifies uncertainty systematically. Not to eliminate chance, but to reason about it precisely.

What You’ll Learn

  1. What probability actually means (long-run frequency vs. single event)
  2. Basic probability rules (addition, multiplication, independence)
  3. Common mistakes (gambler’s fallacy, base rate neglect)
  4. Connection to data analysis and machine learning

Probability: Long-Run Frequency, Not Single-Event Certainty

Experiment: Fair Coin Flips

import random

def flip_coin(n_flips):
    """Simulate coin flips, return proportion of heads."""
    heads_count = sum(random.choice([0, 1]) for _ in range(n_flips))
    return heads_count / n_flips


# Run for different numbers of flips
for n in [10, 100, 1000, 10000]:
    proportion = flip_coin(n)
    print(f"{n:5d} flips: {proportion:.3f} proportion heads")

Output:

   10 flips: 0.400 proportion heads
  100 flips: 0.530 proportion heads
 1000 flips: 0.494 proportion heads
10000 flips: 0.5008 proportion heads

Observation: As n increases, proportion → 0.5 (the “true” probability).

This is the Law of Large Numbers: Long-run frequencies converge to theoretical probabilities.

Basic Probability Rules

Rule 1: Probabilities are between 0 and 1

P(event) ∈ [0, 1]

P(event) = 0  → impossible
P(event) = 1  → certain
P(event) = 0.5 → equally likely to happen or not

Rule 2: Complement Rule

P(not A) = 1 - P(A)

Example:
P(rain tomorrow) = 0.3
P(no rain tomorrow) = 1 - 0.3 = 0.7

Rule 3: Addition Rule (OR)

For mutually exclusive events (can’t both happen):

P(A or B) = P(A) + P(B)

Example: Roll a die
P(roll 1 or 2) = P(1) + P(2) = 1/6 + 1/6 = 2/6

For general events:

P(A or B) = P(A) + P(B) - P(A and B)

Example: Draw a card
P(heart or face card) = P(heart) + P(face) - P(heart face)
                      = 13/52 + 12/52 - 3/52 = 22/52

Rule 4: Multiplication Rule (AND)

For independent events (one doesn’t affect the other):

P(A and B) = P(A) × P(B)

Example: Flip coin twice
P(heads then heads) = P(heads) × P(heads) = 0.5 × 0.5 = 0.25

For dependent events:

P(A and B) = P(A) × P(B|A)

Where P(B|A) = probability of B given A happened

Example: Draw 2 cards without replacement
P(both aces) = P(1st ace) × P(2nd ace | 1st was ace)
             = 4/52 × 3/51
             = 12/2652 ≈ 0.0045

Conditional Probability: P(A|B)

P(A|B) = probability of A, given that B happened

Formula:

P(A|B) = P(A and B) / P(B)

Example: Medical Testing

Scenario:
– Disease prevalence: 1% of population has disease
– Test accuracy: 99% (detects disease 99% of time if present)
– False positive rate: 5% (says positive 5% of time if disease absent)

Question: You test positive. What’s the probability you have the disease?

Intuitive (wrong) answer: “99%” (the test accuracy)

Correct calculation:

# Define probabilities
P_disease = 0.01
P_no_disease = 0.99

P_pos_given_disease = 0.99  # True positive rate
P_pos_given_no_disease = 0.05  # False positive rate

# Total probability of positive test
P_positive = (P_disease * P_pos_given_disease + 
              P_no_disease * P_pos_given_no_disease)

# Bayes' theorem: P(disease | positive test)
P_disease_given_pos = (P_disease * P_pos_given_disease) / P_positive

print(f"P(disease | positive test) = {P_disease_given_pos:.3f}")

Output:

P(disease | positive test) = 0.167

Only 16.7%! Because disease is rare (1%), most positives are false positives.

This is Bayes’ Theorem — the foundation of reasoning under uncertainty.

Common Probability Mistakes

Mistake 1: Gambler’s Fallacy

Wrong reasoning:
– “I’ve flipped heads 5 times in a row”
– “Next flip is more likely to be tails”

Correct: Coin has no memory. P(heads on next flip) = 0.5, regardless of history.

# Simulation: After 5 heads, what happens next?
outcomes_after_5_heads = []

for trial in range(10000):
    flips = [random.choice(['H', 'T']) for _ in range(6)]
    if flips[:5] == ['H'] * 5:
        outcomes_after_5_heads.append(flips[5])

if outcomes_after_5_heads:
    p_heads = outcomes_after_5_heads.count('H') / len(outcomes_after_5_heads)
    print(f"After 5 heads, P(heads on 6th flip) = {p_heads:.3f}")

Output: ~0.5 (no different from any other flip)

Mistake 2: Base Rate Neglect

Scenario:
– 90% of terrorists have characteristic X
– You meet someone with characteristic X
– What’s probability they’re a terrorist?

Wrong: “90%”
Correct: Depends on base rate (how common terrorism is)

If terrorists are 0.001% of population:

P(terrorist | has X) ≈ 0.009 (less than 1%!)

Lesson: P(A|B) ≠ P(B|A). Don’t confuse conditional directions.

Mistake 3: Independence Assumption

Scenario: Weather tomorrow and weather today.

Wrong: P(rain tomorrow) = 20% regardless of today
Correct: P(rain tomorrow | raining today) = 60% (weather is correlated)

Assuming independence when events are dependent causes major errors.

Cross-Domain Connections

Machine Learning: Probabilistic Models

From ML-100.1: What Does Learning Mean?:

# Classification as probability estimation
P(class=cat | image features) = 0.85
P(class=dog | image features) = 0.15

Model outputs probabilities, not certainties.

Data Analysis: Confidence Intervals

From STAT-100.1: What Does Data Tell Us?:

Mean height = 170 cm ± 5 cm (95% confidence)

“95% confidence” = If we repeated sampling 100 times, ~95 intervals would contain true mean.

AI: Bayesian Inference

From AI-300.2: Bayesian Networks:

P(hypothesis | evidence) = P(evidence | hypothesis) × P(hypothesis) / P(evidence)

Update beliefs systematically as evidence accumulates.

Pattern Passport

Pattern Observed: APPROXIMATION & UNCERTAINTY

How It Appears Here:
– Probability = quantified uncertainty (0 to 1)
– Long-run frequency = interpretation via repeated trials
– Conditional probability = updating beliefs given evidence
– Rules = systematic operations on uncertain quantities

Representation:
– Events: Outcomes we’re uncertain about
– Probabilities: Numbers between 0 and 1
– Conditions: P(A|B) = probability given evidence

Transformation Rules:
– Complement: P(not A) = 1 – P(A)
– Addition: P(A or B) = P(A) + P(B) – P(A and B)
– Multiplication: P(A and B) = P(A) × P(B|A)
– Bayes’ Theorem: P(A|B) = P(B|A) × P(A) / P(B)

Assumptions:
– Events have well-defined probabilities
– Repeated trials under same conditions possible (for frequency interpretation)
– Events can be classified as independent or dependent

Failure Conditions:
1. Gambler’s fallacy: Assuming past affects independent future events
2. Base rate neglect: Ignoring prior probabilities
3. Confusion of conditionals: P(A|B) ≠ P(B|A)
4. False independence: Treating correlated events as independent

Related Disciplines:
– ML ML-100.1: Probabilistic predictions
– Statistics STAT-100.1: Confidence intervals
– AI AI-300.2: Bayesian reasoning

Next Learning Steps:
1. STAT-200.2: Conditional Probability — Deep dive into P(A|B)
2. STAT-300.1: Bayesian Inference — Systematic belief updating
3. STAT-300.3: Hypothesis Testing — Decision-making under uncertainty

Summary: Probability Quantifies Uncertainty

Probability = systematic reasoning about uncertainty.

Three key insights:

  1. Long-run frequency: Probabilities are limits of proportions as trials → ∞
  2. Conditional probability: P(A|B) updates beliefs given evidence
  3. Rules are strict: Can’t just “feel” probabilities — must calculate correctly

Common mistakes:
– Gambler’s fallacy (past affects independent future)
– Base rate neglect (ignoring priors)
– Confusing P(A|B) with P(B|A)
– Assuming independence incorrectly

The value: Quantified uncertainty enables rational decisions, statistical inference, and machine learning.

Exercises

  1. Calculate: Two dice. What’s P(sum = 7)? What’s P(sum > 10)?

  2. Conditional: 60% of emails are spam. Spam detector catches 95% of spam, but flags 10% of legitimate email. You receive flagged email. What’s P(actually spam)?

  3. Independence: Events A and B have P(A)=0.4, P(B)=0.5, P(A and B)=0.2. Are they independent?

  4. Simulation: Code the medical testing example. Vary disease prevalence. At what prevalence does P(disease|positive) > 0.5?

Revision History

2026-07-30: Substantially revised to include code demonstrations, medical testing example, common mistakes, cross-domain connections.

2024-11-20: Originally published.

Reproducible Code

All code in this article is available at:
– Code/probability_examples.py — All demonstrations
– validation/test_probability.py — Validation tests

Run tests:

cd Categories/02-Statistics-and-Probability/200.1-What-Does-Chance-Tell-Us
python validation/test_probability.py