AWS, Google, Microsoft and OCI Boost AI Inference Performance with Nvidia Dynamo (blogs.nvidia.com)

NVIDIA announced that its Dynamo software platform — now integrated into managed Kubernetes services from AWS, Google Cloud, Microsoft Azure and Oracle Cloud Infrastructure (and adopted by providers like Nebius) — unlocks production-scale, multi-node (“disaggregated”) inference across NVIDIA Blackwell systems (GB200, GB300 NVL72 and ND GB200-v6). The move follows independent SemiAnalysis InferenceMAX v1 results showing Blackwell’s lead in performance/efficiency and builds on demonstrations like a 1.1M tokens/sec aggregate throughput with 72 Blackwell Ultra GPUs and Baseten’s reported 2× latency and 1.6× throughput gains on long-context code generation without extra hardware. Cloud integrations include AWS EKS, a Dynamo recipe on Google’s AI Hypercomputer, Azure Kubernetes Service support, and OCI Superclusters. Technically, Dynamo enables splitting inference pipelines (prefill vs. decode, routing, expert shards for MoE) across independently optimized GPUs to avoid resource bottlenecks that occur when both phases run on the same device. NVIDIA Grove, a new API inside Dynamo, lets teams declare high-level requirements (e.g., node counts, placement, interconnect affinity) and automatically coordinates scaling, placement and dependencies across clusters. The outcome: higher throughput, lower TCO and practical multi-node support for large reasoning and MoE models at enterprise scale, making distributed inference a viable production pattern for demanding, real-time AI services.