Why This Matters
Given: 5 houses with (size, price) data
Task: Predict price of 1200 sq ft house
Linear regression finds the best line through data points.
Simple, interpretable, surprisingly powerful—and the foundation for understanding all machine learning.
What You’ll Learn
- What “best fit line” actually means (minimizing errors)
- How to find it (closed-form solution + gradient descent)
- When linear regression works and when it fails
- Interpreting coefficients (what do the numbers mean?)
The Problem: Find the Best Line
Training Data
# Size (sq ft), Price ($1000s)
data = [
(500, 150),
(1000, 250),
(1500, 350),
(2000, 450),
(2500, 550)
]
Goal
Find line: price = w₀ + w₁ × size
Where:
– w₀ = intercept (base price)
– w₁ = slope (price per sq ft)
Question: Which line is “best”?
Defining “Best”: Minimize Squared Errors
Candidate Lines
# Line 1: price = 0 + 0.2 × size
# Predictions: 100, 200, 300, 400, 500
# Errors: -50, -50, -50, -50, -50
# Line 2: price = 50 + 0.2 × size
# Predictions: 150, 250, 350, 450, 550
# Errors: 0, 0, 0, 0, 0 ← Perfect!
Line 2 is better—zero error.
Loss Function: Mean Squared Error (MSE)
def mse(w0, w1, data):
"""Calculate mean squared error."""
total_error = 0
for size, actual_price in data:
predicted = w0 + w1 * size
error = predicted - actual_price
total_error += error ** 2
return total_error / len(data)
# Test candidates
print(f"Line 1 MSE: {mse(0, 0.2, data):.2f}")
print(f"Line 2 MSE: {mse(50, 0.2, data):.2f}")
Output:
Line 1 MSE: 2500.00
Line 2 MSE: 0.00
“Best” = line with minimum MSE.
Finding the Best Line: Closed-Form Solution
Mathematical Derivation
For y = w₀ + w₁×x, optimal parameters are:
w₁ = Σ((xᵢ - x̄)(yᵢ - ȳ)) / Σ((xᵢ - x̄)²)
w₀ = ȳ - w₁×x̄
where x̄, ȳ are means of x and y
Python Implementation
def linear_regression_closed_form(data):
"""Find best fit line using closed-form solution."""
# Extract x and y
x_values = [size for size, _ in data]
y_values = [price for _, price in data]
# Calculate means
x_mean = sum(x_values) / len(x_values)
y_mean = sum(y_values) / len(y_values)
# Calculate w1 (slope)
numerator = sum((x - x_mean) * (y - y_mean)
for x, y in zip(x_values, y_values))
denominator = sum((x - x_mean) ** 2 for x in x_values)
w1 = numerator / denominator
# Calculate w0 (intercept)
w0 = y_mean - w1 * x_mean
return w0, w1
w0, w1 = linear_regression_closed_form(data)
print(f"Best line: price = {w0:.2f} + {w1:.4f} × size")
print(f"MSE: {mse(w0, w1, data):.2f}")
Output:
Best line: price = 50.00 + 0.2000 × size
MSE: 0.00
Perfect fit! (Because data is exactly linear)
Alternative: Gradient Descent
Why Gradient Descent?
- Closed-form requires matrix inversion (expensive for large data)
- Gradient descent scales better
- Same approach used in deep learning
Algorithm
def gradient_descent(data, learning_rate=0.0001, iterations=1000):
"""Find best fit using gradient descent."""
w0, w1 = 0, 0 # Start with guesses
n = len(data)
for _ in range(iterations):
# Calculate gradients
grad_w0 = 0
grad_w1 = 0
for size, actual_price in data:
predicted = w0 + w1 * size
error = predicted - actual_price
grad_w0 += error
grad_w1 += error * size
grad_w0 /= n
grad_w1 /= n
# Update parameters
w0 -= learning_rate * grad_w0
w1 -= learning_rate * grad_w1
return w0, w1
w0, w1 = gradient_descent(data)
print(f"Gradient descent: price = {w0:.2f} + {w1:.4f} × size")
Output: Same as closed-form (converges to same solution).
Interpreting Coefficients
What w₁ (Slope) Means
w₁ = 0.2
Interpretation: "Each additional sq ft increases price by $200"
Actionable insight from the model!
What w₀ (Intercept) Means
w₀ = 50
Interpretation: "Base price (size=0) is $50,000"
Sometimes makes sense, sometimes not (can’t have 0 sq ft house).
Predictive Use
def predict_price(size, w0, w1):
return w0 + w1 * size
print(f"1200 sq ft house: ${predict_price(1200, w0, w1):.1f}k")
print(f"1800 sq ft house: ${predict_price(1800, w0, w1):.1f}k")
Output:
1200 sq ft house: $290.0k
1800 sq ft house: $410.0k
When Linear Regression Fails
Failure 1: Nonlinear Relationship
# True relationship: price = size²
nonlinear_data = [
(10, 100),
(20, 400),
(30, 900),
(40, 1600)
]
# Linear fit will have high error
w0, w1 = linear_regression_closed_form(nonlinear_data)
print(f"MSE: {mse(w0, w1, nonlinear_data):.2f}") # High!
Solution: Feature engineering (add size² as feature) or use nonlinear model.
Failure 2: Outliers
data_with_outlier = [
(500, 150),
(1000, 250),
(1500, 350),
(2000, 450),
(2500, 550),
(1200, 10) # Outlier!
]
# Line gets pulled toward outlier
w0, w1 = linear_regression_closed_form(data_with_outlier)
# All predictions now worse
Solution: Outlier removal, robust regression (absolute error instead of squared).
Failure 3: Extrapolation
# Trained on houses 500-2500 sq ft
# Predict 10,000 sq ft mansion
predict_price(10000, w0, w1) # $2050k
Problem: Model has no data about mansions. Linear trend may not hold.
Lesson: Don’t extrapolate far beyond training data.
Cross-Domain Connections
Statistics: Correlation
From STAT-100.3: Correlation vs. Causation:
# Correlation coefficient r related to w₁
# r close to ±1 → strong linear relationship
# r close to 0 → weak/no linear relationship
Mathematics: Optimization
From MATH-300.2: Optimization:
Finding best line = solving optimization problem:
minimize MSE(w₀, w₁)
Uses calculus (derivatives) to find minimum.
Code: Gradient Descent
From CODE-100.1: Why Code?:
Gradient descent is an algorithm—precise steps executed repeatedly until convergence.
Pattern Passport
Pattern Observed: APPROXIMATION & UNCERTAINTY
How It Appears Here:
– Model = Linear function approximating data
– Loss = Quantified prediction error
– Learning = Finding parameters that minimize loss
– Generalization = Model works on new, unseen data
Representation:
– Parameters: w₀ (intercept), w₁ (slope)
– Prediction: ŷ = w₀ + w₁×x
– Error: (ŷ – y)²
Transformation Rules:
– Gradient descent: w ← w – α × ∇Loss(w)
– Closed-form: w = (XᵀX)⁻¹Xᵀy (matrix form)
Assumptions:
– Relationship is approximately linear
– Errors are independent, normally distributed
– No multicollinearity (for multiple features)
Failure Conditions:
1. Nonlinearity: True relationship not linear
2. Outliers: Squared error sensitive to extreme values
3. Extrapolation: Predictions far outside training range
4. Insufficient data: Few points, high noise
Related Disciplines:
– Statistics STAT-100.3: Correlation and causation
– Math MATH-300.2: Optimization theory
– Code CODE-100.1: Algorithmic implementation
Next Learning Steps:
1. ML-100.3: Classification — Predicting categories
2. ML-200.1: Regularization — Preventing overfitting
3. ML-200.3: Feature Engineering — Better inputs → better models
Summary: The Foundation of Supervised Learning
Linear regression = finding best line through data.
Three key insights:
- “Best” means minimizing error (typically squared error)
- Two ways to find it: Closed-form (exact) or gradient descent (iterative)
- Interpretation matters: Coefficients have real-world meaning
When it works:
– Linear relationships
– Continuous outcomes
– Sufficient data, low noise
When it fails:
– Nonlinear patterns
– Heavy outliers
– Extrapolation beyond training range
The value: Simple, interpretable, foundational. Understand this, and deep learning becomes “linear regression with fancy transformations.”
Exercises
-
Code it: Implement both closed-form and gradient descent. Verify they give same result.
-
Visualize: Plot data points and best-fit line. Add prediction for new point.
-
Multiple features: Extend to
price = w₀ + w₁×size + w₂×bedrooms. How does math change? -
Error metrics: Compare MSE, MAE (mean absolute error), RMSE. When is each appropriate?
Revision History
2026-07-30: Substantially revised with code implementations, failure modes, interpretations.
2024-07-10: Originally published.
Reproducible Code
Available at:
– Code/linear_regression.py
– validation/test_regression.py
cd Categories/04-Machine-Learning/100.2-Linear-Regression
python validation/test_regression.py