Neural audio codecs: how to get audio into LLMs (kyutai.org)

The article explains why “speech LLMs” still lag behind text LLMs and shows how neural audio codecs are the practical bridge to put audio into and out of large language models. Naive approaches that predict raw audio sample-by-sample (WaveNet-style, μ‑law 256 quantization at 16 kHz) produce babble and screeches: sequence lengths are enormous (10s of audio = 160k samples) so even a 151M‑parameter transformer with 2,048 context tokens only sees ~128 ms of audio and takes minutes to synthesize 10 seconds. The author ran open-source experiments on a 1,000‑hour Libri-Light subset (≈53 GB) to demonstrate these limits. The fix is to compress audio into discrete, low-rate token streams with neural codecs (e.g., VQ‑VAE / quantized autoencoders) so an LM predicts compressed-token continuations and a decoder reconstructs waveforms. Key techniques: vector quantization with straight‑through gradient estimation to enable end‑to‑end training, a commitment loss to keep embeddings near codebook centers, and multi-stage/residual quantizers (product quantization) to scale codebook capacity without prohibitive memory. The article also introduces Mimi, Kyutai’s codec (used by Moshi and Sesame’s CSM), and argues this path is essential to capture prosody, emotion, sarcasm and other speech phenomena that text-wrapping systems miss—making real speech understanding and efficient real-time generation feasible.