Why smarter models won't lead to AI co-workers (usize.github.io)

In a recent article, the challenges of developing AI co-workers using current large language models (LLMs) were critically examined. The primary issue highlighted is that LLMs lack the ability to discern who is communicating with them, treating all tokens equally regardless of their origin. This architectural shortcoming complicates interactions, especially in multi-user environments like Slack, where authority and context can vary significantly between users. Simply making the models smarter won’t solve this identity ambiguity; it requires a foundational shift in how user identity is integrated into the model's architecture. To address this, the concept of "Instructional Segment Embedding" is proposed, which would allow models to understand and differentiate between user inputs based on their identity. This involves creating a parallel embedding channel for identity information and bridging it with existing delegated authorization systems. By mapping authenticated principals directly into model embeddings, a structured approach can enable LLMs to recognize and respond appropriately to various users. This dual-layer solution could pave the way for more effective and secure AI agents that can operate in complex organizational hierarchies, thus moving closer to realizing the potential of AI co-workers.