Show HN: VoxaDub – Open-source AI video dubbing with speaker-aware voices (github.com)

VoxaDub has announced the release of an open-source AI video dubbing tool that utilizes speaker-aware voices, enabling users to create dubbed videos in multiple languages while maintaining the original speaker's voice. The dubbing process involves transcribing the video dialogue, translating it using the Hy-MT2 model, and synthesizing the speech with the VoxCPM2 model, ensuring that the dubbed audio aligns with the original scene. Notably, VoxaDub offers both local-first offline dubbing and experimental live broadcast dubbing capabilities, allowing users to process videos entirely on their own hardware or through cloud platforms like Colab. This development is significant for the AI/ML community as it combines advanced transcription, translation, and voice synthesis technologies into a seamless package that enhances multilingual content accessibility. The tool supports features such as duration-aware synchronization and optional lip-sync, which improve the overall quality of the dubbed videos. With community-driven speaker diarization for identifying speaker voices and various model options available, VoxaDub positions itself as a versatile solution for content creators looking to broaden their audience reach without sacrificing the integrity of the original performances. This open-source approach encourages collaboration and further innovation in the growing field of audiovisual translation.