Computer Vision

Teach the Camera

My eyes are a picture brain called MobileNet. Press Download to fetch it (about 15 MB), then teach me!

Robo's eyes aren't downloaded yet. It's the MobileNet vision model: about 15 MB (1.5 MB engine + 14 MB of learned numbers). Nothing downloads until you press the button.
Robo needs your camera to see.
Pictures never leave your computer.
Teach me first!
Add a few examples to at least two groups.

What's happening inside

1 · What Robo sees
2 · Turned into 1,280 numbers
3 · 5 closest examples
4 · They vote
Teach me first!

Robo's idea map

Every example becomes a dot. Things that look alike to Robo land close together. The star is what Robo sees right now.

How Robo decides

1Robo already learned to see shapes and colors from a million photos. It turns any picture into a list of 1,280 numbers — its "description".
2When you record, Robo saves the description of each example. That's the learning: it takes less than a second!
3To guess, Robo finds the few examples most like what it sees now (up to 5), and they vote. The closest ones get the loudest votes!
Examples
0
Guess speed
– ms

How it works

Make groups

Name the things you want Robo to tell apart, like "thumbs up" and "just me".

Show examples

Hold the button and wiggle around. More examples, from different angles, make Robo smarter.

Closest friends vote

Robo compares what it sees to all your examples. The most similar ones decide the answer.

Try to trick it

Change the light, move far away, or show something new. When does Robo get confused? Teach it more!

What just happened?

For grown-ups: what's really going on
  • This is transfer learning, the same trick as Google's Teachable Machine. A MobileNet v2 CNN pretrained on ImageNet (1.28M images, 1000 classes) runs in your browser via TensorFlow.js.
  • We drop its final classification layer and use the penultimate activations (mobilenet.infer(img, true)) as an embedding: a 1280-dimensional feature vector that captures visual concepts.
  • The classifier is a hand-written k-nearest neighbours using cosine similarity on L2-normalised embeddings, with k = min(5, smallest class size). Each neighbour votes with weight exp(20·(sᵢ − s_max)), a softmax over similarities; confidence is the normalised vote share. The "self-quiz" is leave-one-out accuracy on (up to 80 of) your own examples. No gradient descent is needed, which is why "training" is instant.
  • The idea map is a live PCA (top two principal components, found by power iteration with deflation) of all stored embeddings. Clusters that separate in 2D are usually easy for KNN; overlapping ones are where it gets confused.
  • Failure modes worth exploring: background/lighting shortcuts (the model may learn "dark room" rather than "hand"), class imbalance, and too few varied examples. All images stay on your device.