New 'Access to Justice' Benchmark: Leading AI Models Miss Critical Legal Details (epyhia.com)

Vals AI and NextLadder Ventures have introduced the Access to Justice Benchmark, revealing that leading AI models struggle to address critical legal details in 252 simulated legal-aid scenarios. None of the 23 models evaluated reached the 80% accuracy threshold for being a reliable first point of contact for low-income Americans facing civil legal issues, with Alibaba's Qwen 3.8 Max scoring the highest at 76.4%. The benchmark specifically assessed models on their ability to navigate common legal challenges such as evictions and child custody, reflecting a significant shift from prior benchmarks that mainly focused on corporate legal work. The findings are significant for the AI/ML community as they highlight gaps in the practical application of legal AI tools, especially their ability to guide users to local resources like legal aid programs. While models generally performed well in providing legal explanations, they fell short, achieving only 30-50% accuracy in directing users to specific local resources. The results underscore a persistent “justice gap,” where low-income individuals lack access to adequate legal help, emphasizing the need for AI systems to not only understand legal principles but also connect individuals with practical resources for their legal needs.