Hibiki-Zero:real-time and multilingual speech translation model (kyutai.org)

Hibiki-Zero has been announced as a cutting-edge real-time and multilingual speech translation model capable of translating from French, Spanish, Portuguese, and German to English. Building on its predecessor, Hibiki, this model incorporates an innovative reinforcement learning training methodology that enhances the efficiency of synthetic data creation, reducing latency while maintaining high audio quality and accurate voice transfer. The model's flexibility allows it to adapt seamlessly to different languages with varied grammatical structures, making it significant in advancing real-time translation technologies. Hibiki-Zero's architecture features a decoder-only framework with 3 billion parameters, capable of generating audio tokens at a constant framerate while modeling both source and target speech. By leveraging sentence-level alignment instead of the complex word-level alignment previously required, the model presents a more scalable approach to language adaptation, demonstrated by its successful transition to Italian with minimal data. Performance evaluations indicate that Hibiki-Zero achieves superior translation quality and reduced latency compared to prior models, setting a new benchmark for the AI and ML community in the realm of simultaneous speech translation and opening doors for future enhancements in language support and emotional expression during translations.