Never ask a model to do something a deterministic system can do reliably (cacm.acm.org)

In a recent analysis, software engineer Bhaskar Rajbongshi shared valuable insights into the challenges of building trustworthy AI agents, emphasizing the need for a security-driven approach in AI development. His experience with an AI document verification agent revealed a critical flaw: despite appearing competent, the model confidently provided an incorrect calculation without any safeguards. Rajbongshi argues that, similar to platform security, AI systems should be treated as untrusted components that require rigorous validation of their outputs. He advocates for pairing AI generation with deterministic systems for tasks like arithmetic, enhancing reliability and preventing the acceptance of errors masked by fluency. Rajbongshi underscores that the principles of security—including assuming hostile input, implementing least privilege, and maintaining layered defenses—are applicable to AI agent development. He warns against the misconception that AI agents necessitate entirely new strategies, suggesting instead that established security practices should guide their design. Key recommendations include verifying machine outputs against deterministic systems, preserving human corrections in workflows, and continuously measuring system performance against known cases. His insights serve as a reminder that the AI/ML community can greatly benefit from the well-established wisdom of security engineering, ensuring that AI models are robust and trustworthy.