Pick the Clues
The robot can only draw straight lines. Give it smarter clues, like distance from the centre, and the hard puzzle becomes easy. Beware of useless clues!
Right now I only know x and y, and I can only cut with a straight line. Tick some clues and press Train. Which clue solves the puzzle?
class Aclass Bpainted = the robot's answer
Challenges
Stuck: on Bullseye with only x and y, see the accuracy stay under 65%.
Bullseye solved: reach 90% accuracy.
Switch to Corners and reach 90%. Which clue is the key?
Junk hurts: take a good set of clues and add junk clues until accuracy drops by 3 points or more compared to the same set without junk.
What just happened?
- With only x and y, the robot can cut the picture with a straight line. A bullseye or a checkerboard cannot be split by one line, so it guesses.
- The clue distance from the centre turns the bullseye into a simple one-number question: is it small or big? The clue x times y does the same for the corners.
- Making new clues from the ones you have is called feature engineering. Good clues beat a smarter robot.
- Random junk clues carry no information, but the robot can still use them to memorise the training dots. Held-out accuracy drops, especially with few training points.
- More training points make junk less harmful, but the best answer is still: use fewer, better clues.
For grown-ups: what's really going on
The model is logistic regression trained by batch gradient descent on standardised features with a small L2 penalty (λ = 0.001). Because it is linear in its inputs, it can only draw a straight boundary in feature space. With the right derived feature the boundary becomes curved in the original (x, y) picture.
- Bullseye: label is
x² + y² < 0.75². The featurer = √(x²+y²)(orx²plusy²together) makes it linearly separable. Corners: label isx·y > 0, and the single interaction featurex·yseparates it perfectly apart from 3% label noise. - Importance here is each clue's share of the total absolute weight. Features are standardised so weights are comparable. This is a simple measure; correlated clues share credit in confusing ways.
- The four junk features are uniform random numbers independent of the label. With
ntraining points anddfeatures, a model can overfit by fitting noise. Held-out accuracy (400 fresh points) exposes it; training accuracy usually does not. - Related ideas: polynomial features, the kernel trick (do this implicitly), neural networks (learn their own features), and feature selection (L1 penalty, mutual information, permutation importance).