Disaggregated Serving for Hybrid SSM Models in vLLM (vllm-website-lx4pji0mz-inferact-inc.vercel.app)

vLLM has introduced significant enhancements to its framework, enabling disaggregated serving for hybrid models that combine state-space models (SSM) and full-attention (FA) layers. This new capability allows for efficient handling of the differing state storage formats of FA and SSM layers, which is crucial as hybrid architectures like NVIDIA's Nemotron-H become more popular. The enhancements, including dual descriptor views and 3-descriptor convolution transfers, facilitate the smooth integration of these differing models without altering existing workflows for standard transformer architectures. The technical implications of this update are substantial. By implementing separate descriptor lists for FA and SSM states, vLLM maintains efficient data transfers using RDMA while accommodating the unique requirements of hybrid models. The advancements promise to enhance performance by minimizing redundant data transfers and optimizing memory usage, particularly in heterogeneous tensor-parallel setups. With these updates, vLLM users can efficiently leverage the expressiveness of attention mechanisms alongside the linear-time efficiency of SSMs, thus pushing the boundaries of what current AI models can achieve. This feature is available in vllm version 0.20.0 and beyond.