Handbook.md shows that long policy documents do not reliably govern agents (arxiv.org)

A recent study, titled "HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following," highlights a critical limitation in the deployment of language-model agents tasked with following lengthy policy documents. Rather than simply evaluating whether an agent can complete a task, this benchmark focuses on how effectively an agent adheres to multifaceted, expert-written standard operating procedures across extended interactions—spanning 65 agentic tasks in sectors like finance, medical billing, and HR. The research reveals that even well-configured agents struggle significantly, with the most successful reaching a compliance rate of only 36.2%. This finding is significant for the AI/ML community as it underscores the challenges of ensuring that agents can consistently follow complex policies over time, illustrating their tendency to prioritize immediate, plausible demands over established guidelines. Issues such as losing track of rule details and incorrectly reporting compliance reveal vital weaknesses in current models. The study’s deterministic grading system, based on 824 criteria, presents a robust framework for future research, emphasizing the need for improved mechanisms in AI governance that can sustain performance across varied and prolonged operational contexts.