The Economics of Open-Weight Inference (data.ornn.com)

A recent study by Ornn Data challenges the prevailing notion that each new NVIDIA GPU generation makes its predecessors obsolete, highlighting the economic value of older GPUs through the lens of open-weight demand. The research reveals that, when utilizing self-hosting on rented hardware, the compute costs for the A100 GPU can drop to as low as $0.12 to $0.35 per million output tokens, making it more cost-effective than the newer H100 under specific workloads. This finding emphasizes that older models can still thrive economically as long as they are fit for appropriate workloads, thus extending their useful life in an evolving ecosystem. This analysis is significant for the AI/ML community as it underscores the viability of older hardware in executing modern AI tasks. It demonstrates that the demand for open-weight models—those with independent deployment capabilities—can shape market dynamics and rental prices, allowing for competitive performance in various applications such as long-running agents and batch evaluations. The insights into GPU occupancy rates and rental prices further illuminate how the economic landscape is shifting, suggesting that the cost-effectiveness of older GPUs can benefit users seeking flexible and budget-conscious solutions in their AI deployments.