Executable governance for AI: turning policy text into runnable tests (arxiv.org)

A new framework called Policy-to-Tests (P2T) has been developed to streamline the process of converting AI policy documents, traditionally written in prose, into machine-readable rules. This initiative is significant for the AI/ML community as it addresses the inefficiencies of manually translating policy into executable rules, a process that can delay important safeguards in real-world applications. By leveraging large language models (LLMs), P2T normalizes policy documents into a compact domain-specific language that includes various clauses, conditions, and evidence requirements, ensuring that generated rules align closely with established human benchmarks. The framework has been tested on various policies, demonstrating a high level of accuracy in rule extraction and inter-annotator agreement. Moreover, it evaluated the impact of these policies by applying HIPAA-derived safeguards to a generative AI agent and measuring its compliance against an unguarded counterpart. This not only highlights the importance of executable governance in enhancing AI safety but also opens up opportunities for reproducibility and further research, as the team has made the codebase, prompts, and rule sets available as open-source resources.