Meta AI announces first AI-powered speech translation system for an unwritten language (venturebeat.com)

Meta AI has unveiled the world’s first AI-powered speech-to-speech translation system for an unwritten language, targeting Hokkien, a predominantly spoken Chinese dialect. This breakthrough is a major leap for AI translation technology, which historically focused on widely written languages, leaving over 40% of oral-only languages underserved. Meta’s universal speech translator (UST) enables real-time bidirectional communication between Hokkien and English speakers—a critical tool for fostering global understanding and inclusion, especially in emerging virtual environments like the metaverse. The system overcomes significant challenges by collecting human-annotated and automatically mined speech data, using novel pseudo-labeling techniques to train on limited resources. It employs a cutting-edge self-supervised approach, converting speech directly into discrete acoustic units without relying on intermediate text representations. The model leverages Mandarin as a linguistic bridge for supervision, using a two-pass decoding mechanism (UnitY) to improve translation accuracy. To benchmark performance, Meta developed a phonetic transcription system and released a large parallel speech corpus, Speech Matrix, which covers 272 language directions with 418,000 hours of data—paving the way for scalable AI translation across numerous oral languages worldwide. Beyond this technical milestone, Meta’s innovation promises to accelerate natural, immersive communication in the metaverse, enabling multilingual interactions in real time with near-human-quality translations. While challenges remain, such as data diversity, computational demand, and privacy-compliant training data collection, advances in unsupervised learning and synthetic data generation hold promise. Meta’s open-sourced tools and datasets invite the broader AI community to build on these foundations, potentially transforming how spoken languages—written or not—connect people in physical and digital spaces alike.