From Evaluation to Guardrails: What We Brought to ACM FAccT 2026 (blog.mozilla.ai)

At the recent ACM Conference on Fairness, Accountability, and Transparency (FAccT) in Montreal, a team presented their tutorial on the "Contextual Evaluation of LLM Guardrails Across Languages and Agentic Systems." This session highlighted the urgent need for robust evaluation mechanisms not only for large language models (LLMs) but also for the guardrails that govern their outputs. As AI safety discussions evolve from merely assessing performance to measuring real-world implications, the tutorial emphasized that the mechanisms designed to filter harmful content should receive equal scrutiny. The team introduced a collaborative evaluation methodology involving 120 scenario pairs related to refugees and asylum, assessed by native-language evaluators, to create concrete guardrail policies that reflect contextual and linguistic nuances. A key finding from their hands-on session was the significant influence of integrated tools, such as search and retrieval, on evaluation outcomes. When tools were employed, the accuracy of the assessments varied greatly depending on the judge LLM used, with some demonstrating better factual verification capabilities than others. This experimentation underscored the necessity for adaptable technological solutions like Mozilla’s open-source any-guardrail, which allows for seamless control over guardrail layers. The implications for the AI/ML community are profound, as the reliance on these dynamic guardrails can enhance both the safety and reliability of AI applications in sensitive fields like humanitarian aid and finance. Future efforts will focus on expanding tool access to refine these systems further across multiple languages, promising a more trustworthy AI landscape.