Target 1: Baseten (www.silares.com)

SAIL, an internal AI lab, recently optimized the Orpheus-TTS text-to-speech deployment on the Baseten platform, achieving a remarkable tenfold increase in concurrency without changing the model's architecture or retraining. Initially, the system handled about 24 real-time connections per NVIDIA H100 GPU, but after implementing system-level enhancements, it now supports 216 concurrent connections while maintaining strict p99 latency and real-time factor constraints. This optimization is significant for the AI/ML community as it demonstrates a cost-effective approach to increasing system throughput, drastically reducing annual accelerator expenses from approximately $1.4 million to $140,000 in a typical deployment scenario. The work highlights the importance of system-level optimizations over traditional model-level techniques, focusing on aspects such as CPU-GPU interaction and pipeline efficiency. SAIL's empirical methodology, which avoids premature optimizations, reveals how performance bottlenecks can be identified and resolved incrementally. By emphasizing a holistic approach to system performance, this initiative delivers insights that can easily extend to other text-to-speech architectures and inference setups, reinforcing the need for broader strategies that encompass both architectural and operational aspects of AI deployment.