Why brains need a bend
Two robot brains, same dots, same size. One has no bend. One has a bend. Pick a shape and guess: can the bendy one draw it?
No bend
With a bend
Oops-meter over time (lower is better)
Stacking with no bend: still one cut
Many layers, but without a bend they melt into one straight cut. Here's the left brain squashed into one rule, live:
One neuron, one bend
Challenges
1. Deeper, still no bend
3 layers, no bend, versus ReLU on the circle. Does more depth save the left brain?
2. Sigmoid sinks, ReLU swims
A 4-layer net with sigmoid on the left and ReLU on the right. Watch the left oops-meter barely move.
3. Dead ReLU
Learn a while, then push every neuron's mood far below zero. Can it ever wake up?
How it works
Add up
Every neuron multiplies inputs by knobs and adds them: that is a straight-line kind of math.
Bend it
Then the bend squashes or clips the total. ReLU chops off negatives. Sigmoid squeezes to 0 to 1.
Stack
Bent neurons give the next layer new shapes to mix: corners, rings, swirls.
Learn
The robot nudges each knob using the slope of every bend. Flat bend means no nudge.
What just happened?
- Without a bend, any number of layers is just one straight cut.
- With a bend, the same brain painted rings, corners and swirls.
- Sigmoid goes flat at the ends, so in deep nets the blame signal fades away.
- A ReLU neuron stuck below zero stays silent and stops learning.
For grown-ups: what's really going on
Two identical MLPs (same seed, data, learning rate 0.5, full-batch gradient descent, sigmoid output with cross-entropy) differ only in the hidden activation.
- Linearity: a composition of linear (affine) maps is one affine map:
W₃(W₂(W₁x + b₁) + b₂) + b₃ = Wx + b. The collapse box multiplies the real weight matrices. The decision boundaryw·x + b = 0is a line, so circle, XOR and spiral data cannot be separated. - Non-linear activations break this. With enough hidden units a one-hidden-layer net with a non-polynomial activation can approximate any continuous function on a compact set: the universal approximation theorem.
- ReLU
max(0, z): cheap, slope 1 for z > 0, gives piecewise-linear boundaries. tanh is zero-centred, range (−1, 1). Sigmoid1/(1+e^−z)has slope at most 0.25. - Vanishing gradients: backprop multiplies one activation slope per layer; with sigmoid each factor is at most 0.25, so early-layer gradients shrink geometrically with depth (the first-layer gradient size is smaller by several orders of magnitude in challenge 2).
- Dying ReLU: if a unit's pre-activation is negative for all inputs its gradient is exactly 0, so its weights never change. Remedies: lower learning rate, leaky ReLU, careful init.
- Step has zero slope almost everywhere, so we train it with a surrogate gradient (a hard-tanh-style slope), a trick used for binary and spiking networks.