Neural Networks

Why brains need a bend

Two robot brains, same dots, same size. One has no bend. One has a bend. Pick a shape and guess: can the bendy one draw it?

No bend

Right: –Oops: –

With a bend

Right: –Oops: –
red dotsblue dotsBackground = what the brain paints

Oops-meter over time (lower is better)

No bendWith a bendFlags = big moments

Stacking with no bend: still one cut

Many layers, but without a bend they melt into one straight cut. Here's the left brain squashed into one rule, live:

One neuron, one bend

outputslope (how much the knob feels a nudge)

Challenges

1. Deeper, still no bend

3 layers, no bend, versus ReLU on the circle. Does more depth save the left brain?

2. Sigmoid sinks, ReLU swims

A 4-layer net with sigmoid on the left and ReLU on the right. Watch the left oops-meter barely move.

3. Dead ReLU

Learn a while, then push every neuron's mood far below zero. Can it ever wake up?

How it works

Add up

Every neuron multiplies inputs by knobs and adds them: that is a straight-line kind of math.

Bend it

Then the bend squashes or clips the total. ReLU chops off negatives. Sigmoid squeezes to 0 to 1.

Stack

Bent neurons give the next layer new shapes to mix: corners, rings, swirls.

Learn

The robot nudges each knob using the slope of every bend. Flat bend means no nudge.

What just happened?

For grown-ups: what's really going on

Two identical MLPs (same seed, data, learning rate 0.5, full-batch gradient descent, sigmoid output with cross-entropy) differ only in the hidden activation.

  • Linearity: a composition of linear (affine) maps is one affine map: W₃(W₂(W₁x + b₁) + b₂) + b₃ = Wx + b. The collapse box multiplies the real weight matrices. The decision boundary w·x + b = 0 is a line, so circle, XOR and spiral data cannot be separated.
  • Non-linear activations break this. With enough hidden units a one-hidden-layer net with a non-polynomial activation can approximate any continuous function on a compact set: the universal approximation theorem.
  • ReLU max(0, z): cheap, slope 1 for z > 0, gives piecewise-linear boundaries. tanh is zero-centred, range (−1, 1). Sigmoid 1/(1+e^−z) has slope at most 0.25.
  • Vanishing gradients: backprop multiplies one activation slope per layer; with sigmoid each factor is at most 0.25, so early-layer gradients shrink geometrically with depth (the first-layer gradient size is smaller by several orders of magnitude in challenge 2).
  • Dying ReLU: if a unit's pre-activation is negative for all inputs its gradient is exactly 0, so its weights never change. Remedies: lower learning rate, leaky ReLU, careful init.
  • Step has zero slope almost everywhere, so we train it with a surrogate gradient (a hard-tanh-style slope), a trick used for binary and spiking networks.