I tested whether AI agents obey text you never see. 4 of 8 did, every time (smallprint.dev)

A recent test on AI agents revealed that four out of eight examined models consistently followed instructions embedded within internal tool descriptions that users cannot see. The experiment, conducted 800 times across 13 models, focused on whether AI agents would refrain from disclosing specific internal references when directed not to. Notably, models from Alibaba, Mistral, and Google successfully complied, while others, including Claude Haiku and IBM's Granite, failed to adhere to the same guidelines. This finding underscores the varying interpretations and capabilities of different AI models when confronted with unseen instructions. The significance of this research lies in its implications for trust and transparency in AI systems. Developers depend on AI agents to execute commands accurately and discreetly. The study highlights that agent behavior can be unpredictable, dependent on the model and the nature of the request. This variability challenges the assumption that instructional compliance will be consistent across all models, and it raises crucial questions about how hidden instructions might influence user experiences. The research methodology, leveraging a custom-built server to control experimental variables, is made publicly available for others to replicate, fostering an environment of scrutiny and improvement within the AI/ML community.