Neural Networks

Backprop, step by step

Here is a tiny brain: 2 inputs, 2 middle neurons, 1 answer. Every wire has a weight. Can you set them so the answer hits the target? Try the sliders!

1. Try it by hand

Output
0
Loss (oops)
0
Your tweaks
0

2. One round, one click at a time

1 Forward pass
2 Measure the oops
3 Backward pass
4 Nudge
Round
0
Output
?
Target
1
Loss
?

Ready

Tip: in the backward pass, tap any wire to see the chain-rule product for that weight.

3. Watch it learn

Green = positive weight Red = negative weight Thicker = bigger Arrows = blame

Snapshots

Snapshots appear at round 1, 10, 100, 1000 and when it's solved. Tap one to time-travel.

Challenges

Each one breaks the network above at the current round. The chart marks the moment so you can see what happens next. Press Auto after!

Pick a challenge, then press Auto in the section above and watch the loss line.

How it works

Forward

Numbers travel left to right. Each neuron adds up its wires and squashes the total.

Oops

Compare the answer with the target. Wrong by a lot means a big oops number.

Blame backwards

Starting at the answer, work out how much each weight helped cause the oops.

Nudge

Move every weight a small step the opposite way from its blame. Then repeat!

What just happened?

For grown-ups: what's really going on

This is backpropagation on a 2-2-1 multilayer perceptron with sigmoid σ(z) = 1/(1+e^−z) in both layers.

  • Loss: squared error L = (o − y)², averaged over the dataset in the "watch it learn" loop (full-batch gradient descent).
  • Chain rule: ∂L/∂w2ⱼ = 2(o−y) · o(1−o) · hⱼ and ∂L/∂w1ᵢⱼ = 2(o−y) · o(1−o) · w2ⱼ · hⱼ(1−hⱼ) · xᵢ. Each factor is one local derivative; the blame signal δ is reused for every weight behind it.
  • Update: w ← w − η ∂L/∂w, with learning rate η. Too big overshoots; too small crawls.
  • Gradient check: the "check my work" table compares analytic gradients with the central finite difference (L(w+ε) − L(w−ε)) / 2ε, ε = 10⁻⁴. They agree to ~10⁻⁹. This is how backprop implementations are unit-tested.
  • Vanishing gradients: σ'(z) ≤ 0.25, and it approaches 0 when |z| is large (saturation). Multiply many small factors and early layers get almost no signal. Dead or saturated units behave similarly.
  • Symmetry: identical initial weights give identical gradients, so hidden units stay clones forever. Random initialization breaks the tie.
  • Why it's efficient: reverse-mode automatic differentiation computes the gradient for all N weights in about the cost of one or two forward passes, instead of N extra passes as in finite differences.
  • Further reading: TensorFlow Playground · 3Blue1Brown, What is backpropagation really doing?