Why This Matters
Deep network: Millions of parameters across dozens of layers.
Question: How do you adjust parameter #347,219 in layer 5 to reduce output error?
Naive answer: Try changing it slightly, see what happens. (Would take billions of years.)
Backpropagation: Efficiently calculate exactly how each parameter affects the error. Updates all parameters in one pass.
This is why deep learning works.
What You’ll Learn
- What backpropagation actually computes (gradients)
- Why it’s efficient (chain rule + caching)
- How gradients flow backward through layers
- What breaks backpropagation (vanishing/exploding gradients)
The Problem: Training Deep Networks
Network Structure
Input (x) → Layer 1 → Layer 2 → ... → Layer N → Output (ŷ)
↓
Loss = (ŷ - y)²
Parameters: w₁, w₂, …, wₙ (weights in each layer)
Goal: Adjust all w to minimize Loss.
Gradient Descent Reminder
From ML-100.2: Linear Regression:
w = w - learning_rate × gradient
Need: gradient of Loss with respect to EVERY parameter.
Challenge: Parameters in early layers affect Loss through many intervening layers.
The Insight: Chain Rule
Simple Example
y = f(g(h(x)))
dy/dx = (df/dg) × (dg/dh) × (dh/dx)
Chain rule: Derivative of composition = product of individual derivatives.
Neural Network Application
Layer 1: z₁ = w₁×x
Layer 2: z₂ = w₂×z₁
Loss: L = (z₂ - y)²
dL/dw₁ = (dL/dz₂) × (dz₂/dz₁) × (dz₁/dw₁)
Key insight: We can compute gradient for w₁ by chaining local gradients backward from loss.
Backpropagation Algorithm: Concrete Example
Network Setup
# Simple 2-layer network
# Input x → Hidden → Output ŷ
def forward_pass(x, w1, w2):
"""Forward pass: compute prediction."""
z1 = w1 * x # Hidden layer (linear)
z2 = w2 * z1 # Output layer
return z1, z2
def loss(y_pred, y_true):
"""Squared error loss."""
return (y_pred - y_true) ** 2
Forward Pass
# Data
x = 2.0
y_true = 8.0
# Initial weights
w1 = 1.0
w2 = 1.0
# Forward
z1, y_pred = forward_pass(x, w1, w2)
L = loss(y_pred, y_true)
print(f"z1 = {z1}") # 2.0
print(f"y_pred = {y_pred}") # 2.0
print(f"Loss = {L}") # 36.0 (big error!)
Backward Pass (Backpropagation)
def backward_pass(x, y_true, w1, w2, z1, y_pred):
"""Compute gradients using chain rule."""
# Gradient of loss with respect to prediction
dL_dy_pred = 2 * (y_pred - y_true) # = -12.0
# Gradient with respect to w2
dy_pred_dw2 = z1 # Partial derivative of w2*z1
dL_dw2 = dL_dy_pred * dy_pred_dw2 # = -12 * 2 = -24
# Gradient with respect to z1 (needed for w1)
dy_pred_dz1 = w2
dL_dz1 = dL_dy_pred * dy_pred_dz1 # = -12 * 1 = -12
# Gradient with respect to w1
dz1_dw1 = x
dL_dw1 = dL_dz1 * dz1_dw1 # = -12 * 2 = -24
return dL_dw1, dL_dw2
dL_dw1, dL_dw2 = backward_pass(x, y_true, w1, w2, z1, y_pred)
print(f"Gradient w1: {dL_dw1}") # -24
print(f"Gradient w2: {dL_dw2}") # -24
Update Weights
learning_rate = 0.01
w1_new = w1 - learning_rate * dL_dw1 # 1.0 - 0.01*(-24) = 1.24
w2_new = w2 - learning_rate * dL_dw2 # 1.0 - 0.01*(-24) = 1.24
# Check: Loss decreased?
z1_new, y_pred_new = forward_pass(x, w1_new, w2_new)
L_new = loss(y_pred_new, y_true)
print(f"Old loss: {L:.2f}") # 36.00
print(f"New loss: {L_new:.2f}") # 20.46 (improved!)
Backpropagation computed exact gradients efficiently!
Why Backpropagation is Efficient
Naive Approach: Finite Differences
def gradient_finite_diff(x, y_true, w1, w2, epsilon=1e-5):
"""Estimate gradient by trying small changes."""
_, y_pred = forward_pass(x, w1, w2)
L = loss(y_pred, y_true)
# Try changing w1
_, y_pred_plus = forward_pass(x, w1 + epsilon, w2)
L_plus = loss(y_pred_plus, y_true)
dL_dw1 = (L_plus - L) / epsilon
# Repeat for w2...
return dL_dw1
Problem: Requires 2 forward passes per parameter.
Cost: O(N) forward passes for N parameters.
Backpropagation: One Backward Pass
Cost: O(1) — single backward pass computes ALL gradients.
For deep networks with millions of parameters, this is the difference between feasible and impossible.
Adding Nonlinearity: Activation Functions
Why Needed
# Linear network
y = w2 * (w1 * x) = (w2 * w1) * x
# Equivalent to single layer!
# Deep linear network = shallow network
Solution: Add nonlinearity between layers.
ReLU Activation
def relu(z):
"""Rectified Linear Unit."""
return max(0, z)
def relu_derivative(z):
"""Derivative of ReLU."""
return 1 if z > 0 else 0
# Forward with ReLU
def forward_with_relu(x, w1, w2):
z1 = w1 * x
a1 = relu(z1) # Activation
z2 = w2 * a1
return z1, a1, z2
# Backward with ReLU
def backward_with_relu(x, y_true, w1, w2, z1, a1, y_pred):
dL_dy_pred = 2 * (y_pred - y_true)
# w2 gradient
dL_dw2 = dL_dy_pred * a1
# Backprop through ReLU
dL_da1 = dL_dy_pred * w2
da1_dz1 = relu_derivative(z1) # 0 if z1<0, else 1
dL_dz1 = dL_da1 * da1_dz1 # Gradient blocked if z1<0
# w1 gradient
dL_dw1 = dL_dz1 * x
return dL_dw1, dL_dw2
Key: ReLU derivative is 0 for negative inputs → gradient doesn’t flow.
When Backpropagation Fails
Failure 1: Vanishing Gradients
# Deep network with sigmoid activation
def sigmoid(z):
return 1 / (1 + exp(-z))
def sigmoid_derivative(z):
s = sigmoid(z)
return s * (1 - s) # Maximum value: 0.25
# Gradient through 10 layers
gradient = 1.0
for layer in range(10):
gradient *= 0.25 # Multiply by sigmoid derivative
print(gradient) # ≈ 0.000001 (vanished!)
Problem: Gradients shrink exponentially with depth.
Solution: ReLU (derivative=1 for positive), residual connections, batch normalization.
Failure 2: Exploding Gradients
# Gradients grow exponentially
gradient = 1.0
for layer in range(10):
gradient *= 2.0 # Large weights
print(gradient) # 1024 (exploded!)
Problem: Parameter updates become huge, training unstable.
Solution: Gradient clipping, careful initialization, lower learning rate.
Failure 3: Dead ReLU
# Neuron outputs negative value
z = -5
a = relu(z) # 0
# Gradient
da_dz = relu_derivative(z) # 0
# No gradient flows → neuron never updates → "dead"
Problem: ReLU neurons can get stuck at 0.
Solution: Leaky ReLU (small negative slope), careful initialization.
Cross-Domain Connections
Mathematics: Chain Rule
From MATH-200.2: Calculus Foundations:
d/dx[f(g(x))] = f'(g(x)) × g'(x)
Backpropagation is automated chain rule application.
Code: Dynamic Programming
From CODE-300.2: Algorithm Design:
Backpropagation caches intermediate results (forward pass) to avoid recomputation (backward pass). Classic dynamic programming.
Pattern Passport
Pattern Observed: COMPOSITION + EFFICIENT COMPUTATION
How It Appears Here:
– Forward pass: Compose layers (y = f₃(f₂(f₁(x))))
– Backward pass: Chain rule to compute all gradients
– Efficiency: One backward pass gets all gradients
Representation:
– Forward: Values flowing forward (x → z₁ → z₂ → y)
– Backward: Gradients flowing backward (dL/dy → dL/dz₂ → dL/dz₁ → dL/dx)
Transformation Rules:
– Chain rule: dL/dw₁ = (dL/dz₂) × (dz₂/dz₁) × (dz₁/dw₁)
– Update: w ← w – α × dL/dw
Assumptions:
– Functions are differentiable
– Gradients exist and are computable
– Floating-point precision sufficient
Failure Conditions:
1. Vanishing gradients: Gradients → 0 in deep networks
2. Exploding gradients: Gradients → ∞
3. Dead neurons: ReLU stuck at 0
4. Non-differentiable functions: Can’t compute gradients
Related Disciplines:
– Math MATH-200.2: Chain rule
– Code CODE-300.2: Dynamic programming
– ML ML-100.2: Gradient descent
Next Learning Steps:
1. AI-100.3: Loss Functions — What networks optimize
2. AI-200.2: Optimization Algorithms — Beyond basic gradient descent
3. AI-300.1: Advanced Architectures — Residual connections, attention
Summary: Efficient Gradient Computation
Backpropagation = efficient gradient computation via chain rule.
Three key insights:
- Chain rule automation: Compose local gradients to get end-to-end gradients
- One pass gets all: Single backward pass computes gradients for all parameters
- Efficiency is critical: Makes training deep networks feasible
When it works:
– Differentiable functions
– Reasonable gradient magnitudes
– Proper initialization
When it fails:
– Vanishing gradients (too deep, bad activation)
– Exploding gradients (large weights)
– Dead neurons (ReLU stuck)
The value: Without backpropagation, deep learning impossible. With it, we train networks with billions of parameters.
Exercises
-
Manual calculation: 3-layer network, x=1, y_true=10, all weights=1. Compute all gradients by hand.
-
Verify numerically: Compare backprop gradients to finite differences. Should match to ~1e-5.
-
Gradient flow: Plot gradient magnitude at each layer in 20-layer network. Observe vanishing.
-
Dead ReLU: Create scenario where ReLU neuron dies. Show gradient=0.
Revision History
2026-07-30: Substantially revised with concrete calculations, failure modes, efficiency analysis.
2024-08-05: Originally published.
Reproducible Code
Available at:
– Code/backprop_example.py
– validation/test_backprop.py
cd Categories/05-AI-and-LLMs/100.2-Backpropagation
python validation/test_backprop.py