Topic 09 · Sequence Models · Audio

Speech-to-Text & Text-to-Speech

Whisper transcribes your voice entirely in your browser. Web Speech API reads any text aloud. No audio leaves your device.

Ready — record or upload audio for transcription

Speech → Text (Whisper)

Whisper-tiny · 39MB · cached after first load

Click to start recording
Record or upload audio to transcribe…

Text → Speech (TTS)

Web Speech API · built into your browser

How Whisper Works — Architecture

1 · Audio preprocessing

Audio is resampled to 16 kHz, split into 30-second chunks, and converted to an 80-channel log-Mel spectrogram — a visual time-frequency representation.

2 · Transformer encoder

A 12-layer Transformer encoder processes the spectrogram using 2D sinusoidal position encodings, learning acoustic representations of phonemes, words, and prosody.

3 · Autoregressive decoder

A Transformer decoder generates tokens one-by-one, attending to the encoder output and all previously decoded tokens. Language detection and timestamps are special tokens.

// Correct way to pass audio to Whisper in the browser const audioCtx = new AudioContext({ sampleRate: 16000 }); const decoded = await audioCtx.decodeAudioData(arrayBuffer); const pcm = decoded.getChannelData(0); // Float32Array at 16kHz const result = await whisper(pcm, { language: 'english', task: 'transcribe' }); // → { text: "Hello from Whisper, running in your browser!" }