Sound & music

Teach the Ear

Computers can't hear like you. They turn sound into a picture, then learn what each kind of sound's picture looks like.

Press a sound button (or turn on your microphone) and watch the sound turn into a picture! Low sounds at the bottom, high sounds at the top.

Sound picture (spectrogram) quiet

Robo's quiz score as it hears more examples

Checkpoints: what Robo's sound memories looked like

Saved as Robo collects examples.

How it works

Sound becomes a picture

The computer splits sound into low, middle and high parts many times a second. Loud parts glow brightly.

Every sound has a shape

A whistle is a thin line up high. A hum is a stripe down low. A clap is a short splash everywhere.

Collect examples

Each recording is saved as a little picture in its box. The big picture is their average.

Find the closest

For a new sound, Robo finds the saved pictures that look most alike, and they vote.

What just happened?

For grown-ups: what's really going on
  • Spectrogram: the Web Audio AnalyserNode runs an FFT (2048 samples) about 60× per second. The magnitudes are pooled into 32 log-spaced bands (80 Hz–8 kHz), roughly like a mel spectrogram.
  • Segmentation: a simple energy-based onset detector (adaptive noise floor) captures ~0.75 s after each onset, so every example is aligned to its start.
  • Features: the clip is pooled to a 32 × 12 time-frequency image, normalised to its own peak (so volume matters less), plus its average spectrum.
  • Classifier: k-nearest neighbours (k = 3, cosine distance, distance-weighted votes). The quiz score is leave-one-out accuracy: each example is classified by all the others.
  • Real systems (keyword spotting, speech recognition like Whisper) feed log-mel spectrograms into CNNs or transformers trained on thousands of hours of audio. Teachable Machine's audio mode uses a pretrained network's embeddings instead of raw pixels, which generalises better than this.