Why Johnny Cant Use Agents: Aspirations vs. Realities with AI Agents (arxiv.org)

Researchers examined the gap between industry hype about “AI agents” and how real users actually experience them. They systematically reviewed 102 commercial agents and found marketed use cases cluster into three buckets—orchestration (coordination and automation), creation (content generation), and insight (analysis and recommendations). Then they ran a usability study with 31 participants who attempted representative tasks on two popular commercial agents, Operator and Manus. While participants were often impressed by what agents could do, the study uncovered consistent usability failures: capabilities that didn’t match users’ mental models, poor transparency about limits, and a lack of meta-cognitive behaviors (self-monitoring, uncertainty signaling, clarification-seeking and error recovery) needed for effective human–agent collaboration. This work matters for AI/ML practitioners because it shows that commercial success and technical capability aren’t enough—agents must be designed around realistic user expectations and collaboration patterns. Technically, the findings point to priorities for future work: clearer capability taxonomies, better uncertainty estimation and explainability, interactive clarification protocols, and evaluation frameworks that measure not just task output but alignment with user mental models and meta-cognitive competence. For product teams, the paper is a call to temper marketing, surface limitations, and invest in interaction designs that scaffold users rather than assuming seamless autonomous competence.