The Open/Closed Problem in AI (blog.mempko.com)

At the recent MLSys conference in Seattle, researchers and industry experts gathered to discuss advancements in machine learning, particularly around large language models (LLMs). A central theme that emerged was the "Open/Closed problem," highlighting the shift in AI hardware from general-purpose CPUs to specialized ASICs that optimize for efficiency in tasks like inference and training. This evolution mirrors a historical trajectory in computing where flexibility has been traded for efficiency, potentially stifling innovative approaches to AI. The concern is that, as hardware becomes more specialized, it embeds assumptions that hinder the pursuit of closed-loop learning, where models could autonomously update their parameters based on new information. The significance of this issue lies in how it may limit the future development of AI systems. While many are focused on enhancing the efficiency of existing open-loop learning models, the real breakthrough may require a radical redesign that combines memory and computational processes, akin to how biological neurons operate. The call to action is clear: there is an urgent need for a new substrate that enables closed-loop learning to flourish before the opportunity to explore this paradigm narrows further due to ongoing hardware specialization. Researchers must rethink their hardware choices to facilitate innovative learning approaches rather than reinforcing the status quo.