High Dimensional Optimization Remains Hard (win-vector.com)

Recent discussions and foundational research confirm that high-dimensional optimization remains inherently challenging, despite some theoretical results suggesting local minima are rare relative to saddle points. Seminal works, including those by Bray and Dean (2007), Dauphin et al. (2014), and Choromanska et al. (2015), highlight the complexity of landscapes like those seen in neural networks and spin glasses. However, these results often rely on strong assumptions—such as statistical independence and specific random field models—that rarely hold in practical applications. Consequently, real-world optimization problems frequently encounter numerous obstacles beyond simple local minima, including flat regions, numeric instabilities, and complex global structures that can trap gradient-based methods. A vivid example is presented in a train scheduling problem that models arrival times as charged particles repelling each other, creating a highly non-convex energy surface with factorially many separated valleys. This structure fundamentally limits gradient-based optimizers, which cannot easily cross the “ridges” formed by ordering constraints without large perturbations. Such difficulties illustrate why high-dimensional spaces do not guarantee easy escape routes from suboptimal regions, as optimizers often become confined to local basins far from the global optimum. The problem’s physical analogy—combining electrostatic repulsion and gravity—underscores that even smooth, bounded objective functions can be intractably complex due to inherent structural symmetries and constraints. This analysis challenges the oversimplified notion that high-dimensional optimization is easier and stresses the importance of understanding global problem geometry. It encourages techniques like staged optimization, where difficult terms are initially suppressed, enabling easier convergence before reintroducing complexity. Such insights emphasize that advancing optimization in AI requires not only better algorithms but also carefully crafted problem formulations acknowledging the nuanced topologies and constraints that arise in practical, structured tasks.