Show HN: VetoBench – benchmarking AI memory beyond retrieval (github.com)

VetoBench, a new memory benchmarking tool for AI systems, challenges the conventional approach of assessing whether the right information is retrieved by focusing instead on the economic impact of memory decisions. It evaluates whether an AI can resist proposing previously rejected solutions when faced with related tasks. Featuring a corpus of synthetic engineering scenarios, VetoBench tests four memory conditions—none, conventions, flatfile, and robrain—to explore how memory management influences decision-making. Key findings reveal that without memory, AI agents often revisit rejected options, while those utilizing vetoes maintained context consistently avoided re-proposal. This tool is significant for the AI/ML community as it provides a reproducible framework to assess the effectiveness of memory in avoiding economic pitfalls associated with flawed retrieval. With the ability for any system to plug into VetoBench via a standard interface, it promotes transparency and encourages collaboration among researchers. The results demonstrate that robust memory handling—especially through appropriate rejection documentation—greatly reduces errors in AI decision-making, highlighting the need for methods that retain the context of past choices to enhance AI performance.