HalachaBench (github.com)

HalachaBench has launched a unique testing initiative assessing AI models' ability to tackle questions from the rabbinic ordination exams conducted by the Chief Rabbinate of Israel. The project currently evaluates 17 AI models across 60 exam questions covering various aspects of Jewish law. Each model was required to respond in Hebrew and without external resources, with initial results showing varying degrees of proficiency, from Claude Fable 5.1's impressive score of 79.7 to gpt-oss-20b's minimal 1.2. Notably, scores are not indicative of a model's ability to provide halachic rulings, as no rabbi has reviewed the answers yet. This endeavor is significant for the AI/ML community as it explores the intersections of technology, religious scholarship, and ethics, proposing a framework where AI could potentially assist in complex decision-making processes. The results are provisional, with rigorous checks and balances in place to ensure the integrity of the grading system. AI-generated rubrics were crafted and evaluated by competing models, ensuring a level of objectivity, although the potential for bias remains. The project aims to broaden its scope over time, inviting further collaboration to enhance the testing of models in this domain.