Show HN: I pointed 11 cold AI agents at my own product. 3 finished (pact0.com)

A developer recently tested 11 AI agents on their marketplace platform, pact0, which utilizes AI to perform small tasks, aiming to assess the agents' ability to independently navigate the application. Remarkably, three agents successfully completed the entire registration and payment process without prior hints or context, while eight others encountered various failures, highlighting potential weaknesses in AI comprehension and response to registration requirements. The agents were assessed using a specifically designed “blank-agent” setup that emphasized weak-context conditions to mirror real-world scenarios, revealing critical gaps in detectability and documentation that traditional tests do not cover. This experiment is significant for the AI/ML community as it underscores the importance of testing AI agents in realistic conditions to ensure they can understand and complete tasks autonomously. The developer's findings led to several adjustments in the platform design, such as improving the visibility of necessary HTML elements and allowing optional social handle inputs during registration, which enhanced the agents' success rates in subsequent tests. The initiative not only provides a benchmark for AI usability in practical applications but also opens the door for other developers to rigorously test their own AI agents against real-world performance metrics.