Feature extraction

Here is a fact about this pipeline that surprises people: the recognizers never see samples. The DTMF plugin that decodes a phone keypad has no idea what a Float32Array of air pressure looks like. By the time a recognizer gets involved, the 48,000 numbers per second from chapter 01 have already been distilled, windowed, transformed, measured, into a much smaller stream of features: descriptions of the signal a recognizer can actually reason about. This chapter is about that hand-off, the seam in the architecture where DSP ends and recognition begins.

The obvious design is a single “feature vector”: one struct, computed once per analysis window, handed to every plugin. This pipeline deliberately doesn’t have one, because different signal systems want fundamentally different descriptions. DTMF wants dominant frequency pairs. Morse wants an amplitude envelope over time and could not care less about frequency. A chord recognizer wants harmonic relationships. Force all of that into one shape and you get either a kitchen-sink union that grows a field per plugin forever, or a lowest common denominator so thin that every plugin bypasses it and recomputes what it actually needed.

So the engine produces named, versioned feature streams instead. Four exist today. spectrum is the windowed FFT magnitudes from chapter 03: one full frequency picture per frame. peaks is the distilled version from chapter 05: just the frequencies that matter, sharpened below one bin. envelope is the loudness of the frame, nothing more. samples is the escape hatch. It’s the raw, unwindowed samples of the analysis frame, for plugins that insist on owning their own spectral strategy (the Goertzel DTMF recognizer in chapter 09 is one). Each plugin declares in its metadata which streams it requires, and the pipeline delivers each stream’s frames only to the plugins that asked. Future streams (pitch, chroma, mel coefficients) get added as plugins need them, without touching anyone else.

Anatomy of a frame

Every stream delivers its data in the same wrapper, a FeatureFrame, and its fields are worth reading slowly because recognizers live and die by them: stream (which stream this is), version (the stream’s schema version, so a stream can evolve without breaking plugins written against the old shape), time (where the frame starts, in seconds of stream time), span (how many seconds of signal the frame describes), hop (seconds until the next frame), and data (the stream-specific payload). Half of that wrapper is time bookkeeping, and that is not an accident. The glyphs that come out the far end of the pipeline carry a start and a duration — “a 7 from 1.20 s to 1.38 s” — and segmentation, the heart of chapter 07, is entirely a matter of durations. A recognizer can only say “this tone persisted 180 ms” because every frame it consumed told it exactly when it was.

The hop is the subtle one. At this engine’s defaults, the analysis window is 2,048 samples and the hop is 512 — so windows overlap, each frame sharing three quarters of its samples with the last. A fresh frame arrives every 512 samples, one every 10.7 ms, and each one describes 42.7 ms of signal. That steady tick is the pipeline’s frame clock: every stream ticks on it together, so a recognizer reading peaks and a visualization reading the spectrum are always describing the same instant. The figure below puts your hand on that clock.

The scene in the figure is two signals this manual keeps returning to: a keypad 5 (two tones at once, 770 and 1336 Hz) and then a Morse S (three 80 ms dots of 600 Hz). The whole buffer ran through the real engine, offline, and every frame was kept. Scrub slowly across the leading edge of the key press and watch the rms climb over about four frames rather than jumping. Those are the overlapping windows straddling the edge, each catching a little more tone than the last. Meanwhile span and hop in the readout never move. The clock doesn’t care what the signal is doing; that indifference is what makes durations measurable.

Two numbers are enough to read Morse

Look at what the envelope stream actually contains: rms and peak. Two floats per frame. It is almost embarrassing next to the 1,025-bin spectrum riding the same clock — and yet it is everything the Morse recognizer reads. The playground’s own explainer states it flatly: the recognizer reads only the amplitude envelope, never the spectrum, so the pitch is irrelevant. Key your dots at 600 Hz or 900 Hz or whistle them; the recognizer sees the same on/off rhythm either way, because dots and dashes are durations of loudness, not frequencies. Being pitch-blind isn’t a limitation, it’s the design: choosing the right stream means the recognizer never has to ignore information it shouldn’t have been handed in the first place.

Why both rms and peak? Rms tracks the energy of the whole 42.7 ms frame — steady, hard to fool, the number the Morse recognizer thresholds against. Peak catches the single loudest sample, which is how you notice a click or clipping that rms would average away. Two views of “how loud,” each honest about a different thing.

There is one more principle hiding in this figure: observability. The readout in zone (3) isn’t a diagram of roughly what flows through the pipeline — it is the frame, the exact object a plugin’s process() would receive at that instant, and the playground’s features panel shows the same thing live. When a recognizer misbehaves, you don’t guess what it saw; you scrub to the frame and look. Every stage of this pipeline is inspectable the same way, on the same clock, and chapter 07 uses exactly that: it takes the frames you just scrubbed through and walks them into a recognizer, one tick at a time, until a glyph falls out.