What Eight Agent Systems Record About What They Did (machinetestimony.org)

A recent conformance census evaluated eight AI agent systems—OMEM, mem0, Graphiti, LangGraph, CrewAI, the OpenAI Agents SDK, AutoGen, and Letta Code—against the Testimony Record specification, which aims to assess how well these systems maintain records of actions and decisions. This is significant for the AI/ML community as it highlights crucial gaps in accountability, traceability, and human oversight within these popular frameworks. The census revealed that while most systems could not provide a clear record of decision approvals, data destruction, or verify the integrity of past records, OMEM uniquely met all conformance requirements, emphasizing the need for better standards in AI accountability. The study reframed its requirements as questions about the stored evidence rather than mere compliance, demonstrating that a system's ability to retain information is critical for true conformance. For instance, most systems did not document who approved actions, raising concerns about human oversight in AI decision-making. The validation process also indicated that merely holding data in an inaccessible format is insufficient for compliance, as missing records cannot be recovered. This census serves as a wake-up call for developers and researchers focused on evolving AI technologies, emphasizing the importance of robust, verifiable systems that genuinely reflect evidence of decision-making processes.