AgingBench: AI Agents Age Too (agingbench.github.io)

AgingBench has been introduced as a novel benchmark to evaluate the reliability of long-lived AI agents, acknowledging that traditional benchmarks assess models only at initialization and overlook the ongoing performance of deployed systems. The significance of AgingBench lies in its focus on the lifespan of AI agents, highlighting that their reliability diminishes over time due to various aging mechanisms that affect their operational effectiveness. This approach recognizes that AI reliability is a property of the entire agent lifecycle, not just its initial state. The framework categorizes the aging of agents into four mechanisms: compression aging, interference aging, revision aging, and maintenance aging. Each mechanism reflects different ways an agent's knowledge and functionality can degrade as it interacts with the environment. The benchmark utilizes advanced tools like temporal dependency graphs and counterfactual probes to diagnose performance issues throughout the memory management processes. Results from extensive tests across diverse scenarios reveal that agent aging is complex, indicating that simple behavioral checks may not fully capture underlying factual degradation. This underscores the necessity for tailored lifespan evaluation and diagnostic techniques to ensure the sustained reliability of AI agents in deployment.