DGX station and "frontier" models, my hunt for answers (www.atcyrus.com)

NVIDIA's DGX Station has generated excitement in the AI/ML community by claiming it supports models up to 1 trillion parameters. However, a closer examination reveals that while it boasts 748GB of coherent memory, this consists of 252GB of high-bandwidth memory (HBM3e) and 496GB of lower-bandwidth LPDDR5X memory. This memory split raises questions about performance when running large models, as the speed benefits of HBM may not translate if workloads extend into the LPDDR5X tier. Reports from researchers at Cornell and other institutions using the DGX Station highlight mixed results in handling large models effectively, complicating the decision for potential buyers. With a price tag of approximately $100,000, the DGX Station is positioned as a serious investment that competes with multi-GPU rigs and cloud services. The primary concern for users is whether the machine can meet the high-speed demands of frontier AI workloads without sacrificing performance due to its unique memory architecture. While early tests indicate promising throughput for specific workloads, the community remains skeptical, seeking concrete benchmarks to fully gauge the machine’s capabilities in a practical setting. As excitement grows, the need for tangible performance data is more critical than ever in determining the viability of the DGX Station for real-world applications.