Serving LLMs on Tenstorrent Hardware: Inside the vLLM TT Plugin (vllm.ai)

Tenstorrent has announced a significant enhancement to its AI infrastructure with the launch of the vLLM TT Plugin, designed to enable the deployment of large language models (LLMs) on Tenstorrent hardware. This plugin automatically registers Tenstorrent devices as a native platform for vLLM, utilizing a similar API to OpenAI, ensuring seamless integration for developers. Notably, the Tenstorrent architecture diverges from traditional GPU frameworks, featuring a unique phase-constrained scheduler and a mesh topology that optimizes data parallelism and sampling, allowing for more efficient model execution. The introduction of this plugin holds considerable implications for the AI/ML community, given its support for multimodal models and various architectures, including Llama, Qwen, and Gemma. The capacity to register models via the TT-prefixed convention enables streamlined integration for improved performance and resource management. Additionally, the design allows models to be hand-tuned for Tenstorrent's mesh architecture, enhancing token efficiency and simplifying parallel execution, a feature that could bolster model throughput and cost-effectiveness. As such, this development not only promotes broader model accessibility on advanced hardware but also positions Tenstorrent as a noteworthy player in the rapidly evolving landscape of AI technology.