StepAudio 3 Gen Technical Report (arxiv.org)

The recently released technical report on StepAudio 3 Gen introduces an innovative audio generation model capable of zero-shot text-to-speech (TTS), voice design, and a fusion of various audio types within a single framework. This model stands out by using a discrete autoregressive approach to directly model audio through residual vector quantization (RVQ) tokens, moving away from the commonly used diffusion Transformer paradigm. The StepAudio Tokenizer efficiently quantizes audio at 12.5 Hz in a shared code space, ensuring a harmonious integration of both semantic and waveform-level features. Significantly, StepAudio 3 Gen encapsulates three essential design principles aimed at enhancing audio generation while preserving the capabilities of large language models. Through interference-aware progressive pretraining, an RVQ Adaptor for multi-codebook representations, and discrete autoregressive modeling, the system achieves state-of-the-art performance in TTS and voice design. This model not only excels in spoken applications but also exhibits robust capabilities in generating music, sound effects, and other audio types, making it a versatile tool for the AI/ML community. The availability of audio samples demonstrates its practical application and potential impact on audio generation technologies.