Show HN: Fusion-runtime – self-hosted voice agents, STT+LLM+TTS in one process (github.com)

A new self-hosted voice agent runtime called Fusion-runtime has been announced, capable of running speech-to-text (STT), a large language model (LLM), and text-to-speech (TTS) processes simultaneously on a single machine. This cohesive integration allows for real-time interaction, enabling responses to start playback while they are still being generated, resulting in a processing time of approximately 490 ms after completing a turn. The system can process 127 tokens per second and can handle interruptions mid-sentence, making conversations more fluid and natural. The significance of Fusion-runtime lies in its streamlined architecture and ease of deployment. Users can quickly set up a voice agent with just a few commands using Python, with model configurations handled safely to prevent exposure of sensitive information. It supports various models from well-known platforms like Hugging Face and offers a browser client for direct interaction. The production capability demonstrated on an RTX 3090 reveals efficient performance, with options for scalability. This advancement not only simplifies the development of voice assistants but also supports commercial use, as the project adheres to permissive licensing for its default models.