Pure C, CPU-only inference with Mistral Voxtral Realtime 4B speech to text model (github.com)

The Voxtral Realtime 4B model has been released as a pure C implementation for fast and dependency-free inference, allowing seamless audio transcription using Mistral AI's advanced architecture. This implementation operates solely with the C standard library, featuring two notable backends: Metal Performance Shaders (MPS) for Apple Silicon, which offers rapid performance, and a BLAS backend for Intel systems, though it operates at reduced speeds due to the conversion of BF16 weights to FP32. The design enables real-time streaming with low latency and supports audio input directly from standard input, enhancing its versatility for various applications. This development is significant for the AI/ML community as it democratizes access to powerful speech-to-text capabilities, enabling broader experimentation and integration without reliance on heavy frameworks or libraries. By providing a self-contained Python reference implementation alongside the C code, it simplifies the learning curve for developers unfamiliar with the complex internals of models like vLLM. Key features include a chunked audio processing approach for efficient memory management, support for unlimited audio lengths via a rolling KV cache, and a well-structured streaming API for real-time token generation, making it a robust tool for both developers and researchers in the field.