CSCI 8945 · Audio
State-of-the-Art Audio AI: Case Study Resources
Five models every student of generative audio should study. This page collects the key papers, code, demos, and explainers for each model covered in our case study lectures. The models span audio compression, speech synthesis, speech recognition, and music generation—chosen because each has a published paper, open code, and represents a distinct technical paradigm.
EnCodec
Meta's neural audio codec compresses audio into discrete tokens using a convolutional encoder-decoder with residual vector quantization (RVQ). It is the shared representation layer that VALL-E, F5-TTS, and MusicGen all build on top of. Understanding EnCodec is prerequisite to understanding the generation models.
EnCodec
Compresses 24kHz mono or 48kHz stereo audio at bitrates from 1.5 to 24 kbps, outperforming traditional codecs (Opus, EVS) at equivalent bitrates. The discrete token sequences it produces became the de facto input representation for language-model-based audio generation.
VALL-E
Microsoft's landmark model reframed text-to-speech as language modeling over EnCodec tokens. A 3-second voice sample is enough for zero-shot cloning. The first RVQ level is generated autoregressively (capturing content and identity), while remaining levels are generated non-autoregressively (acoustic detail). This paradigm influenced nearly every subsequent codec-based TTS system.
VALL-E
Trained on 60K hours of English speech (LibriLight). Demonstrated that treating TTS as “next-token prediction on audio codes” could achieve natural prosody and speaker similarity from a 3-second prompt—a capability that previously required minutes of reference audio and fine-tuning.
VALL-E 2
Follow-up that claimed the first human-parity result on zero-shot TTS benchmarks. Introduced repetition-aware sampling to fix the looping artifacts of the original, and grouped code modeling for efficiency. Worth reading alongside the original to see how the paradigm matured.
F5-TTS
A fully non-autoregressive approach that uses flow matching with a Diffusion Transformer (DiT) to generate speech in parallel. Trained with a simple “fill in the middle” objective—masking a portion of the mel spectrogram and learning to reconstruct it—which naturally supports zero-shot voice cloning and speech editing. Contrasts sharply with VALL-E's autoregressive token-by-token generation.
F5-TTS
“A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching.” Achieves competitive quality with a simpler training pipeline than autoregressive alternatives. The non-autoregressive generation means the entire utterance is produced in a fixed number of denoising steps rather than token-by-token, offering a different speed/quality tradeoff.
Whisper
OpenAI's encoder-decoder transformer for automatic speech recognition (ASR). Unlike the generation models above, Whisper operates on mel spectrograms (not codec tokens) and maps audio → text. Its key lesson: a standard architecture trained on 680K hours of weakly supervised web audio can match or beat specialized ASR systems across languages without any fine-tuning.
Whisper
Multitask model that handles transcription, translation, language identification, and timestamp prediction in a single architecture. Available in sizes from tiny (39M params) to large-v3 (1.5B params). The “just scale up the data” result was a turning point for the ASR field and a case study in how weak supervision at scale competes with careful data curation.
MusicGen & Suno
MusicGen is Meta's open-source, single-stage transformer for text-to-music generation, operating over EnCodec tokens with a clever delay pattern that handles multiple codebook streams in one forward pass. Suno is the leading commercial system, producing full songs with vocals, but has no published technical paper.
MusicGen
“Simple and Controllable Music Generation.” Generates high-quality instrumental music conditioned on text descriptions or melody input. The delay pattern—offsetting each codebook stream by one step—lets a single transformer model multiple parallel token streams without the multi-stage cascade that earlier systems (like MusicLM) required. Text conditioning uses T5 embeddings.