Together AI Compiler for Inference (www.together.ai)

Together AI has announced the launch of the Together Compiler for Inference (TKCC), an innovative solution designed to optimize AI inference workloads that vary widely across models, hardware, and user behaviors. With the ability to serve trillions of tokens daily, TKCC utilizes high-throughput agents to continually optimize diverse workloads, achieving up to 2.6 times higher transactions per second (TPS) across leading open-source models. It sets new benchmarks for low-latency performance, with impressive TPS figures on models like MiniMax M3.1-Flash and DeepSeek v4.1-Flash. The significance of TKCC lies in its ability to tackle the complex, dynamic nature of AI inference, which has become increasingly fragmented due to varying architectures and workloads. By employing a "persistent engines" paradigm that minimizes downtime during experimentation and a custom cost model for selecting efficient parallelism strategies, TKCC dramatically enhances optimization speed and accuracy. This not only outperforms existing systems but also extends its applicability beyond language models to other domains, such as weather modeling and biology. The introduction of TKCC marks a pivotal advancement in developing adaptable AI systems that can efficiently respond to continuously evolving traffic patterns.