The Evolution of GPUs: How Floating-Point Changed Computing (www.dell.com)

Floating-point arithmetic — the move from fixed-point binary to a sign×mantissa×2^exponent representation — unlocked the wide dynamic range and precision modern computing needs. FP32 (IEEE‑754 single precision: 1 sign, 8 exponent, 23 fraction bits) can represent roughly ±3.4×10^38 with ~7 decimal digits of precision, while FP64 (double: 1 sign, 11 exponent, 52 fraction bits) provides much greater precision for scientific simulation. Lighter formats like FP16 (1/5/10), BF16 (1/8/7) and NVIDIA’s TF32 (19‑bit hybrid with FP32-like exponent and a reduced mantissa) trade precision for throughput and memory efficiency — a critical design choice for ML where approximate matrix arithmetic is often acceptable. That tension between range, precision, and parallel throughput is why GPUs and specialized units like NVIDIA Tensor Cores transformed AI/ML. GPUs expose massive parallelism and hardware mechanisms (fused multiply‑add, tensor units) that accelerate dense linear algebra used in training and inference. Modern accelerators (A100/H100) support FP64/FP32/FP16/BF16/TF32 to match workloads; edge devices (Jetson Orin NX) emphasize FP16 for real‑time processing. For practitioners, the implication is clear: choose numeric formats and hardware to balance accuracy, speed, and memory — BF16/TF32 or FP16 for fast, large‑scale training and inference; FP64/FP32 for precision‑sensitive science — with Tensor Cores and fused operations delivering the practical performance gains that drive today’s AI breakthroughs.