Draw the Box
Telling a circle from a square is easy. Finding where it is, and how big, is a different job. Draw a box around each toy, then watch a robot learn to do it too.
Here is a toy table! I will name a toy. Drag a box around it as tightly as you can. I score you with IoU: how much your box and the true box overlap.
Part 1 · Draw the box
Part 2 · Watch it learn
Challenges
Make the tables harder with the buttons. The robot keeps training on whatever you pick, and each win puts a flag on the chart. A win means average IoU of 0.50 or more.
How it works
Cut into cells
The table becomes an 8 by 8 grid. Every cell gets the same little brain.
Each cell guesses
"Is a toy here?", how far its middle is, how wide and tall, and which shape.
Compare with the truth
The robot sees how far its guesses are from the real boxes.
Nudge and repeat
Tiny nudges on fresh tables, again and again, until boxes fit snugly.
What just happened?
- Finding a toy means two answers: what it is and where it is (a box).
- IoU scores a box: shared area divided by total area. Perfect is 1.0, no touching is 0.
- The robot split the table into cells, and each cell guessed a box, a shape and how sure it was.
- Nobody told it where toys are. It nudged its numbers to make its boxes overlap the true ones more.
- Big toys got several cells shouting at once. Merging duplicates keeps just the surest box.
- The sureness slider is a trade: too strict misses toys, too relaxed invents ghost boxes.
For grown-ups: what's really going on
Image classification outputs one label per image. Object detection outputs a variable-size set of (class, bounding box, confidence) triples. This lesson trains a tiny YOLO-style, single-shot detector in plain JavaScript on procedurally generated 64x64 scenes.
- IoU (Jaccard index) = area(A ∩ B) / area(A ∪ B). Detections are usually counted as correct at IoU ≥ 0.5.
- Grid cells (YOLO): the image is an 8x8 grid. Each cell outputs an objectness logit, a box (centre offset from the cell centre, width and height through a sigmoid) and 3 class logits. The same small network (169 inputs, 40 tanh units, 8 outputs) is shared across all cells. Its input is a fine 12x12 and a coarse 5x5 crop of a colourfulness map around the cell. Real detectors use anchor boxes and a CNN backbone instead.
- Bounding-box regression: box numbers are trained with squared error; objectness with binary cross-entropy (positives weighted 2x); class with softmax cross-entropy. Cells whose centre lies near an object's centre are all positives (like FCOS centre sampling), so big objects get duplicate predictions. Optimiser: Adam, batches of 8 fresh scenes.
- Non-max suppression: sort by confidence, keep a box, drop any remaining box with IoU > 0.4 against it, repeat.
- The held-out score here is the mean over true objects of the best IoU among shown boxes (0 if missed). Real benchmarks use mAP: sweep the confidence threshold, trace the precision-recall curve per class, take the area under it, and average over classes (and over IoU thresholds for COCO mAP@[.5:.95]). Moving the sureness slider here walks along that precision-recall trade-off.
Further reading: You Only Look Once (Redmon et al., 2016) · FCOS (Tian et al., 2019) · CS231n detection lecture notes.