GPU Implementation of Second-Order Linear and Nonlinear Programming Solvers (arxiv.org)

This arXiv paper surveys and evaluates GPU implementations of second‑order optimization solvers—chiefly pivoting‑free interior‑point methods—for large, sparse linear and nonlinear programs. Building on recent GPU‑accelerated direct sparse linear solvers (e.g., cuDSS), the work compares different Karush–Kuhn–Tucker (KKT) system formulations that are amenable to block‑dense, pivoting‑free factorizations on GPUs, and outlines practical ways to compute sparse Jacobians and Hessians on GPU hardware. Numerical experiments show strong scalability and common speedups exceeding 10× versus comparable CPU solvers when targeting medium numerical precision, demonstrating GPUs’ potential to substantially shorten solve times on large instances. For the AI/ML community this matters because many model fitting, constrained learning and control problems rely on large structured QPs/NLPs where second‑order methods are desirable for robustness and convergence. The paper highlights technical tradeoffs: pivoting‑free schemes and specialized KKT factorizations enable highly parallel GPU execution but can limit numerical robustness and high‑precision accuracy; efficient sparse Jacobian/Hessian assembly (coloring, AD strategies) and GPU memory/communication remain bottlenecks. Overall, the survey maps practical pathways and current limitations for bringing mature second‑order optimization into GPU‑native ML pipelines, pointing to where solver design, precision strategies, and hardware‑aware sparse linear algebra must evolve next.