AI inference costs dropped up to 10x on Nvidia's Blackwell (venturebeat.com)

Nvidia recently announced significant reductions in AI inference costs, with leading providers reporting savings of 4x to 10x per token when utilizing the Blackwell platform alongside optimized software stacks and open-source models. This dramatic decrease is crucial for the AI/ML community as it enables companies across various industries, including healthcare and gaming, to scale their AI applications from pilot phases to widespread deployment. For instance, the healthcare company Sully.ai achieved a 90% reduction in inference costs by automating tasks that previously required extensive manual input, while Latitude slashed its gaming platform costs by 75% through advanced model configurations. The cost reductions stem from combining Blackwell hardware improvements with low-precision formats, like NVFP4, which allow for more efficient computation, and optimized software integrations that enhance throughput. Key factors affecting performance range from precision format adoption to specific model architectures such as mixture-of-experts (MoE), which benefit from Blackwell's unique capabilities. As organizations assess their infrastructure choices, they must consider workload characteristics and evaluate multiple configurations to achieve optimal cost efficiency and performance, balancing the trade-offs between provider offerings and operational complexities.