New names for the same old failure modes (appsdonewright.com)

An AI engineer's experience navigating a new AI course reveals the ongoing challenges within AI/ML systems, despite advancements in design and terminology. In this case, the engineer encounters “agentic tool orchestration,” where models autonomously select tools and process information, but the underlying issues of managing dependencies and decision-making remain. The engineer highlights the unsettling reality of switching models mid-task—a change that complicates accountability and consistency, as different models may yield varying outcomes from the same prompts, including unintended actions like issuing refunds without clarity on decision ownership. Moreover, the exploration of retrieval-augmented generation (RAG) sheds light on the complexities of information retrieval and permissions in AI. While the system can generate coherent responses based on relevant data, it raises critical concerns about privacy and the context of information sourcing. The interaction between automated models and human oversight underscores the need for robust evaluation mechanisms, as failure modes persist, albeit in new forms. This experience illustrates that while AI tools may appear more sophisticated, the fundamental challenges of user control, accountability, and ethical AI remain pressing issues for the community.