Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM (developer.nvidia.com)

A recent tutorial showcased how to serve the NVIDIA GLM-5.3 NVFP4 model, consisting of 753 billion parameters, across two NVIDIA DGX Stations using vLLM. This implementation highlights the use of tensor parallelism to split model weights and computations between the two stations, enabling effective handling of models too large for a single GPU. With each activated token routing to one of the 40 billion experts available, this configuration utilizes both stations over a high-speed ConnectX-8 RDMA connection, pushing performance to 607.8 tok/s. The significance of this deployment lies in its ability to maintain all model weights in GPU memory without CPU weight offload, allowing for efficient operation under heavy loads. Technical advancements such as sparse activation, mixed-precision quantization, and maximum GPU memory utilization enhance computational efficiency. This setup not only illustrates the potential for running frontier-scale AI models locally but also serves as a baseline performance benchmark for future projects in the AI/ML community, reinforcing the importance of distributed systems in handling sophisticated models.