When AI Is Your Pastor: Benchmark for Theological Triage and Pastoral Guidance (fideai.org)

Fide AI has announced the release of FMG-Bench, a benchmark designed to evaluate large language models (LLMs) in providing theological triage and pastoral guidance. This public benchmark consists of a corpus with 120 base scenarios, each with 37 perturbation variants, aimed at testing how well models address complex faith-based questions, many of which involve critical pastoral care considerations. The benchmark assesses 14 advanced models and reveals an average performance improvement of nearly 4 points under guided conditions, with pastoral application scenarios showing the most significant gains in safety and escalation appropriateness. This development is significant for the AI/ML community as it introduces a structured framework for evaluating AI's responses to sensitive theological inquiries, emphasizing the importance of context in pastoral situations. The findings indicate that using guided instructions not only increases the models' accuracy but also their stability when faced with variations in prompt structure. Notably, the benchmarks focus on distinguishing between primary, secondary, and tertiary doctrinal questions, suggesting a nuanced approach is essential for effective pastoral and moral guidance. However, the creators stress that FMG-Bench is a measurement tool and not an endorsement of AI as pastoral authority, marking an important intersection of AI technology with theological discourse.