NVIDIA DGX Spark In-Depth Review: A New Standard for Local AI Inference (lmsys.org)

NVIDIA’s new DGX Spark is a compact, desktop-friendly “mini-DGX” that brings a coherent, unified-memory architecture to local AI inference. Built around a GB10 Grace Blackwell Superchip (20 CPU cores) and a GPU delivering up to 1 PFLOP sparse FP4 tensor performance, the Spark packs 128 GB of unified LPDDR5x memory (≈273 GB/s) and dual QSFP ConnectX-7 links (200 Gb/s aggregate). Two Sparks can be clustered and—according to NVIDIA—handle up to ~405B parameters in FP4. The system targets developers and researchers who need DGX-class convenience for prototyping, on-device inference, and experiments in memory-coherent GPU architectures rather than raw peak TFLOPs. Benchmarks show the trade-offs: the unified memory lets the Spark load very large models (e.g., Llama 3.1 70B ran at 803 tps prefill / 2.7 tps decode), and it shines on smaller models with efficient batching (Llama 3.1 8B: ~7.9k tps prefill, decode scaling to ~368 tps at batch 32). But limited memory bandwidth is the main bottleneck—full-size Blackwell/Ada GPUs remain ~4× faster on 20B workloads. Notably, software techniques like speculative decoding (EAGLE3) in SGLang can yield ~2× end-to-end speedups, mitigating bandwidth limits. With stable thermals, USB‑C external power, Dockerized SGLang/Ollama support and OpenAI‑compatible APIs, DGX Spark positions itself as an elegant, efficient platform for local model serving, edge research, and offline coding assistants.