From CPU Transparency to GPU Complexity – The Performance Engineering Frontier (harvard-edge.github.io)

Over the past month the discussion shifted from CPU transparency to the far harder problem of GPU performance: the question is no longer whether GPUs matter (they power every modern model) but whether LLMs can meaningfully optimize them. Unlike CPUs, GPUs demand explicit parallelism management, careful memory-hierarchy orchestration, and bespoke kernel design across a deep software stack (Tensor Cores, HBM, cuDNN/TensorRT, mixed precision). Early AI-assisted tools (e.g., Google’s ECO for CPU) exposed what’s possible on more predictable hardware; GPUs introduce a qualitatively different search space where multi-turn RL and other sophisticated approaches are starting to show promise but face production realities—latency, numerical precision trade-offs, and determinism—that bench research often omits. The stakes are high: manual GPU tuning is scarce, slow, and locked into NVIDIA’s CUDA ecosystem, creating a “CUDA moat” that inhibits alternative accelerators despite hardware innovation (TPUs, ROCm, new ASICs). Historical lessons (IBM’s Cell) show that raw throughput is useless if the programming model is prohibitive—an opening where LLMs could excel by systematically navigating complex data movement and scheduling. If LLMs can automate kernel-level optimization and cross-platform porting, they’d democratize performance engineering, break vendor lock-in, and unlock orders-of-magnitude gains; if not, GPUs remain a bottleneck with too few experts to meet demand.