Data & Features

Pick the Clues

The robot can only draw straight lines. Give it smarter clues, like distance from the centre, and the hard puzzle becomes easy. Beware of useless clues!

Right now I only know x and y, and I can only cut with a straight line. Tick some clues and press Train. Which clue solves the puzzle?

class Aclass Bpainted = the robot's answer

Challenges

Stuck: on Bullseye with only x and y, see the accuracy stay under 65%.
Bullseye solved: reach 90% accuracy.
Switch to Corners and reach 90%. Which clue is the key?
Junk hurts: take a good set of clues and add junk clues until accuracy drops by 3 points or more compared to the same set without junk.

What just happened?

For grown-ups: what's really going on

The model is logistic regression trained by batch gradient descent on standardised features with a small L2 penalty (λ = 0.001). Because it is linear in its inputs, it can only draw a straight boundary in feature space. With the right derived feature the boundary becomes curved in the original (x, y) picture.

  • Bullseye: label is x² + y² < 0.75². The feature r = √(x²+y²) (or x² plus y² together) makes it linearly separable. Corners: label is x·y > 0, and the single interaction feature x·y separates it perfectly apart from 3% label noise.
  • Importance here is each clue's share of the total absolute weight. Features are standardised so weights are comparable. This is a simple measure; correlated clues share credit in confusing ways.
  • The four junk features are uniform random numbers independent of the label. With n training points and d features, a model can overfit by fitting noise. Held-out accuracy (400 fresh points) exposes it; training accuracy usually does not.
  • Related ideas: polynomial features, the kernel trick (do this implicitly), neural networks (learn their own features), and feature selection (L1 penalty, mutual information, permutation importance).