Computer Vision

Draw the Box

Telling a circle from a square is easy. Finding where it is, and how big, is a different job. Draw a box around each toy, then watch a robot learn to do it too.

Here is a toy table! I will name a toy. Drag a box around it as tightly as you can. I score you with IoU: how much your box and the true box overlap.

Part 1 · Draw the box

Press, drag and let go. Green = where your box and the true box overlap.

Part 2 · Watch it learn

The robot's boxes on a table it never trained on. Tap a toy to ask Robo about it.

Challenges

Make the tables harder with the buttons. The robot keeps training on whatever you pick, and each win puts a flag on the chart. A win means average IoU of 0.50 or more.

How it works

Cut into cells

The table becomes an 8 by 8 grid. Every cell gets the same little brain.

Each cell guesses

"Is a toy here?", how far its middle is, how wide and tall, and which shape.

Compare with the truth

The robot sees how far its guesses are from the real boxes.

Nudge and repeat

Tiny nudges on fresh tables, again and again, until boxes fit snugly.

What just happened?

For grown-ups: what's really going on

Image classification outputs one label per image. Object detection outputs a variable-size set of (class, bounding box, confidence) triples. This lesson trains a tiny YOLO-style, single-shot detector in plain JavaScript on procedurally generated 64x64 scenes.

  • IoU (Jaccard index) = area(A ∩ B) / area(A ∪ B). Detections are usually counted as correct at IoU ≥ 0.5.
  • Grid cells (YOLO): the image is an 8x8 grid. Each cell outputs an objectness logit, a box (centre offset from the cell centre, width and height through a sigmoid) and 3 class logits. The same small network (169 inputs, 40 tanh units, 8 outputs) is shared across all cells. Its input is a fine 12x12 and a coarse 5x5 crop of a colourfulness map around the cell. Real detectors use anchor boxes and a CNN backbone instead.
  • Bounding-box regression: box numbers are trained with squared error; objectness with binary cross-entropy (positives weighted 2x); class with softmax cross-entropy. Cells whose centre lies near an object's centre are all positives (like FCOS centre sampling), so big objects get duplicate predictions. Optimiser: Adam, batches of 8 fresh scenes.
  • Non-max suppression: sort by confidence, keep a box, drop any remaining box with IoU > 0.4 against it, repeat.
  • The held-out score here is the mean over true objects of the best IoU among shown boxes (0 if missed). Real benchmarks use mAP: sweep the confidence threshold, trace the precision-recall curve per class, take the area under it, and average over classes (and over IoU thresholds for COCO mAP@[.5:.95]). Moving the sureness slider here walks along that precision-recall trade-off.

Further reading: You Only Look Once (Redmon et al., 2016) · FCOS (Tian et al., 2019) · CS231n detection lecture notes.