Why I'm still bearish on LLMs after Navier-Stokes (dank.systems)

In a recent analysis, a tech commentator challenges the prevailing optimism surrounding large language models (LLMs), particularly in the context of their recent achievements, such as progress in solving Navier-Stokes equations. The author argues that current frontier models still require substantial human oversight and guardrails, limiting their autonomy and effectiveness in knowledge work. Despite the impressive demonstrations from AI labs, many businesses continue to hire lower-tier software engineers, indicating a disconnect between the capabilities of LLMs and the practical requirements of various tasks. The analysis highlights the critical issue of reward hacking, underscoring that effective use of LLMs necessitates rigorous specification by domain experts—an expensive and specialized endeavor that is often lacking. It notes that while some firms may leverage LLMs for narrow, clearly defined tasks, most businesses face structural barriers that hinder full adoption of autonomous AI. As a result, industries that require comprehensive specifications or can tolerate continual oversight will likely remain reliant on traditional methods and more affordable AI solutions. This skepticism points to ongoing challenges in integrating LLMs, suggesting that their true potential may not be realized without significant advancements in task specification and oversight methodologies.