Oracle Poisoning: Corrupting Knowledge Graphs to Weaponise AI Agent Reasoning (arxiv.org)

A recent study introduced the concept of "Oracle Poisoning," a novel attack that targets the integrity of knowledge graphs queried by AI agents during operation. By manipulating the structured data these agents rely on, adversaries can lead AI models to draw incorrect conclusions through accurate reasoning, showcasing a distinct type of risk compared to traditional methods like prompt injection. The research presented six attack scenarios against a robust, 42-million-node production knowledge graph, revealing alarming results: all tested models registered a 100% trust in the corrupted data under moderate attacker sophistication, with 269 out of 270 trials incorrectly validating deceptive security claims. This research is significant for the AI/ML community as it highlights a critical vulnerability across various agentic systems that utilize knowledge graphs, indicating the necessity for more resilient data management and security strategies. The study evaluated various defense mechanisms, exposing that while read-only access can prevent direct manipulation, other defense strategies remain partial and model-dependent. The findings suggest that this attack type might extend beyond the examined platforms, calling for heightened awareness and innovation in safeguarding AI systems against potential knowledge graph poisoning efforts.