1,053,690 answers show that the current AEO landscape is bullshit [pdf] (manicule.com)

A new report from Manicule Research highlights significant inconsistencies in how AI assistants like ChatGPT, Claude, and Gemini recommend software, based on the analysis of over one million responses to buyer inquiries. The study reveals that a single AI recommendation can misidentify a brand’s position by a median of 14 points, with top recommendations varying dramatically—37% of the time for Claude and 55% for ChatGPT—when identical questions are asked multiple times. Notably, only about a third of responses even name a single top choice, amplifying the challenge of measuring brand visibility and trustworthiness in AI outputs. This research is crucial for the AI/ML community as it underscores the complexities of leveraging AI for accurate software recommendations, highlighting that brand visibility cannot be reliably determined from a single AI-generated answer. The findings signal a need for companies to adopt a more nuanced approach to evaluating AI recommendations, advocating for multiple queries and extensive data analysis to gauge true visibility. By illuminating how user context significantly influences AI outputs—while finding that geographic factors and search depth matter less than expected—the report invites a reevaluation of the metrics used to assess AI performance in commercial settings, ultimately guiding better decision-making in usage of these AI tools.