Why This Matters
You show a child five examples:
– “apple” + picture of apple
– “banana” + picture of banana
– …
Now show a new picture. The child says “apple!” correctly.
What just happened? The child didn’t memorize—they extracted a pattern that generalizes to new examples.
That’s learning. And machine learning is making this process explicit, measurable, and systematic.
What You’ll Learn
By the end of this article, you’ll understand:
- Learning = finding patterns that generalize beyond training data
- The fundamental trade-off: memorization vs. generalization
- How to measure whether learning actually happened
- What breaks learning (insufficient data, wrong assumptions, overfitting)
The Core Idea: Pattern Extraction
Problem: Predict House Prices
Training data (what we observe):
Size (sq ft) | Price ($1000s)
-------------|---------------
500 | 150
1000 | 250
1500 | 350
2000 | 450
2500 | 550
Question: What will a 1200 sq ft house cost?
Approach 1: Memorization (Not Learning)
def predict_memorized(size, training_data):
"""Look up exact match in training data."""
for train_size, train_price in training_data:
if size == train_size:
return train_price
return None # Not in training data!
predict_memorized(1200, training_data) # None - failed!
Problem: Memorization doesn’t generalize. Can only answer for exact training examples.
Approach 2: Pattern Extraction (Learning!)
Observation: Price ≈ 100 + 0.2 × Size
def predict_learned(size):
"""Use discovered pattern."""
return 100 + 0.2 * size
predict_learned(1200) # 340 - reasonable estimate!
predict_learned(1800) # 460 - works for new sizes!
This is learning: We found a pattern (linear relationship) that works for unseen data.
The Learning Process: Concrete Example
Step 1: Choose a Model Family
Model family = class of patterns you’ll search.
For house prices, we choose linear models:
price = w₀ + w₁ × size
Where w₀ and w₁ are parameters we need to find.
Step 2: Define a Loss Function
Loss = how wrong our predictions are.
Mean Squared Error (MSE):
Loss = average of (prediction - actual)²
def calculate_loss(w0, w1, training_data):
"""Calculate how bad current parameters are."""
total_error = 0
for size, actual_price in training_data:
predicted = w0 + w1 * size
error = (predicted - actual_price) ** 2
total_error += error
return total_error / len(training_data)
# Try random parameters
calculate_loss(0, 0.1, training_data) # High loss (bad)
calculate_loss(100, 0.2, training_data) # Low loss (good!)
Step 3: Optimize Parameters
Learning = minimize loss.
def train_linear_model(training_data, learning_rate=0.0001, iterations=1000):
"""Find best parameters using gradient descent."""
w0, w1 = 0, 0 # Start with guesses
for _ in range(iterations):
# Calculate gradients (how to improve)
grad_w0 = 0
grad_w1 = 0
for size, actual_price in training_data:
predicted = w0 + w1 * size
error = predicted - actual_price
grad_w0 += error
grad_w1 += error * size
# Update parameters
w0 -= learning_rate * grad_w0
w1 -= learning_rate * grad_w1
return w0, w1
w0, w1 = train_linear_model(training_data)
print(f"Learned pattern: price = {w0:.1f} + {w1:.4f} × size")
Output:
Learned pattern: price = 50.0 + 0.2000 × size
We found the pattern automatically!
The Critical Test: Does It Generalize?
The Train-Test Split
Training data = data used to find pattern
Test data = new data never seen during training
# Split data
train = [(500, 150), (1000, 250), (1500, 350)]
test = [(2000, 450), (2500, 550)]
# Train on training data
w0, w1 = train_linear_model(train)
# Test on new data
def evaluate(w0, w1, data):
"""Calculate loss on held-out data."""
predictions = [w0 + w1 * size for size, _ in data]
actuals = [price for _, price in data]
errors = [(pred - actual)**2 for pred, actual in zip(predictions, actuals)]
return sum(errors) / len(errors)
train_loss = evaluate(w0, w1, train)
test_loss = evaluate(w0, w1, test)
print(f"Training loss: {train_loss:.2f}")
print(f"Test loss: {test_loss:.2f}")
If test_loss ≈ train_loss: Model generalizes! ✓
If test_loss >> train_loss: Model memorized! ✗
When Learning Fails: Three Failure Modes
Failure 1: Insufficient Data
# Only 2 training examples
sparse_train = [(500, 150), (2500, 550)]
# Many patterns fit equally well:
# price = 50 + 0.2 × size ✓
# price = 100 + 0.18 × size ✓
# price = 0 + 0.22 × size ✓
# But they give different predictions for new data!
Problem: Not enough constraints to nail down the right pattern.
Failure 2: Wrong Model Family
# True relationship is quadratic
true_data = [
(1, 10),
(2, 40),
(3, 90),
(4, 160)
] # price = 10 × size²
# But we use linear model
# Best linear fit: price ≈ 50 × size
# Still has high error because relationship ISN'T linear
Problem: Your model family doesn’t contain the true pattern.
Failure 3: Overfitting (Memorization)
# 5 training points, but use 5-parameter polynomial
# Can fit training data PERFECTLY
# But wiggles wildly between points - doesn't generalize!
# Example:
train = [(1, 2.1), (2, 4.0), (3, 5.9), (4, 8.1), (5, 9.9)]
# Linear fit (2 parameters): Train loss = 0.04, Test loss = 0.05 ✓
# 4th-degree polynomial (5 parameters): Train loss = 0.00, Test loss = 15.3 ✗
Problem: Model is too flexible for amount of data. Fits noise, not signal.
Cross-Domain Connections
Mathematics: Optimization
Learning = solving an optimization problem:
Find w that minimizes Loss(w)
Uses calculus (gradients) from MATH-300.2: Optimization.
Statistics: Inference
Learning connects to STAT-200.2: Maximum Likelihood:
Minimizing squared error ≡ maximizing likelihood under Gaussian noise assumption.
Code: Algorithmic Implementation
Gradient descent is an algorithm—precise steps from CODE-100.1: Why Code?.
Data Pipelines
Real learning requires DATA-200.3: Data Pipelines: cleaning, splitting, validation.
Pattern Passport
Pattern Observed: APPROXIMATION & UNCERTAINTY
How It Appears Here:
– Learning = finding a simplified pattern in noisy data
– Generalization = pattern works on new, unseen examples
– Overfitting = memorizing noise instead of extracting signal
– Validation = measuring performance on held-out data
Representation:
– Parameters (w): Numeric encoding of learned pattern
– Loss function: Quantifies how wrong predictions are
– Training data: Examples used to find pattern
– Test data: Independent check that pattern generalizes
Transformation Rules:
– Gradient descent: Iteratively improve parameters to reduce loss
– Train-test split: Separate data for learning vs. validation
– Cross-validation: More robust generalization estimate
Assumptions:
– Training data is representative of test data
– Model family contains (approximately) the true pattern
– Sufficient data to constrain parameters
– Loss function aligns with actual goals
Failure Conditions:
1. Insufficient data: Multiple patterns fit equally well
2. Wrong model family: True pattern not representable
3. Overfitting: Model too complex for available data
4. Distribution shift: Test data differs from training data
Related Disciplines:
– Mathematics MATH-300.2: Optimization theory
– Statistics STAT-200.2: Maximum likelihood estimation
– Code CODE-100.3: Algorithmic implementation
– Data DATA-200.3: Data pipelines
Next Learning Steps:
1. ML-100.2: Linear Regression — Specific learning algorithm deep-dive
2. ML-100.3: Classification — Learning for categorical outcomes
3. ML-100.4: Overfitting — Detailed analysis of memorization vs. generalization
Summary: Learning is Pattern Extraction
Machine learning = systematically finding patterns that generalize.
Three key insights:
- Learning ≠ Memorization: Must work on new, unseen data
- Generalization is the test: Training accuracy doesn’t matter if test accuracy is poor
- Trade-offs everywhere: Model complexity vs. data amount, bias vs. variance, interpretability vs. accuracy
What breaks learning:
– Not enough data to constrain the pattern
– Wrong model family (can’t represent true relationship)
– Overfitting (memorizing noise)
– Distribution shift (test data differs from training)
The value: Learning makes prediction systematic, measurable, and improvable. Turn intuition into algorithms.
Exercises
-
Generalization test: Given training loss = 0.05 and test loss = 0.04, has the model learned? What about train = 0.01, test = 0.50?
-
Model complexity: You have 10 training examples. Should you use a 2-parameter linear model or a 10-parameter polynomial? Why?
-
Pattern vs. memorization: Write code that memorizes training data vs. code that extracts a pattern. Show they differ on new examples.
-
Loss function design: For house price prediction, why use squared error instead of absolute error? When might absolute error be better?
Revision History
2026-07-30: Substantially revised to include concrete code examples, explicit failure modes, cross-domain connections. Added train-test split demonstration.
2024-04-15: Originally published.
Reproducible Code
All code in this article is available at:
– Code/learning_examples.py — All worked examples
– validation/test_learning_basics.py — Automated tests
Run tests:
cd Categories/04-Machine-Learning/100.1-What-Does-Learning-Mean
python validation/test_learning_basics.py