Backprop, step by step
Here is a tiny brain: 2 inputs, 2 middle neurons, 1 answer. Every wire has a weight. Can you set them so the answer hits the target? Try the sliders!
1. Try it by hand
2. One round, one click at a time
Ready
Tip: in the backward pass, tap any wire to see the chain-rule product for that weight.
3. Watch it learn
Snapshots
Challenges
Each one breaks the network above at the current round. The chart marks the moment so you can see what happens next. Press Auto after!
How it works
Forward
Numbers travel left to right. Each neuron adds up its wires and squashes the total.
Oops
Compare the answer with the target. Wrong by a lot means a big oops number.
Blame backwards
Starting at the answer, work out how much each weight helped cause the oops.
Nudge
Move every weight a small step the opposite way from its blame. Then repeat!
What just happened?
- The network guessed by passing numbers forward, then measured how wrong it was.
- The blame flowed backwards: wires that mattered more got more blame.
- Each weight moved a little, so the next guess is a bit better. Repeat many times and it learns XOR.
- Starting at zero, huge steps, or saturated neurons can make learning stall.
For grown-ups: what's really going on
This is backpropagation on a 2-2-1 multilayer perceptron with sigmoid σ(z) = 1/(1+e^−z) in both layers.
- Loss: squared error
L = (o − y)², averaged over the dataset in the "watch it learn" loop (full-batch gradient descent). - Chain rule:
∂L/∂w2ⱼ = 2(o−y) · o(1−o) · hⱼand∂L/∂w1ᵢⱼ = 2(o−y) · o(1−o) · w2ⱼ · hⱼ(1−hⱼ) · xᵢ. Each factor is one local derivative; the blame signalδis reused for every weight behind it. - Update:
w ← w − η ∂L/∂w, with learning rateη. Too big overshoots; too small crawls. - Gradient check: the "check my work" table compares analytic gradients with the central finite difference
(L(w+ε) − L(w−ε)) / 2ε, ε = 10⁻⁴. They agree to ~10⁻⁹. This is how backprop implementations are unit-tested. - Vanishing gradients:
σ'(z) ≤ 0.25, and it approaches 0 when |z| is large (saturation). Multiply many small factors and early layers get almost no signal. Dead or saturated units behave similarly. - Symmetry: identical initial weights give identical gradients, so hidden units stay clones forever. Random initialization breaks the tie.
- Why it's efficient: reverse-mode automatic differentiation computes the gradient for all N weights in about the cost of one or two forward passes, instead of N extra passes as in finite differences.
- Further reading: TensorFlow Playground · 3Blue1Brown, What is backpropagation really doing?