WebAccessBench: Digital Accessibility Reliability in LLM-Generated Websites [pdf] (conesible.de)

The recent introduction of WebAccessBench marks a significant advancement in assessing the digital accessibility of websites generated by large language models (LLMs). This novel benchmark evaluates the accessibility quality and conformance to WCAG standards under realistic prompting conditions, providing a comprehensive comparison of older and newer LLM models. The findings reveal that while older models may produce fewer overall errors, they exhibit a higher concentration of errors per DOM element, indicating that they may not necessarily be more reliable. In contrast, newer flagship models, although generating more total errors, demonstrate improved error density when normalized by DOM structure. This research underscores the potential gaps in LLM performance regarding accessibility, which is crucial for developers aiming to create inclusive web interfaces. The study's multi-metric approach highlights that simply counting errors isn't enough; understanding their distribution across the code is vital for risk assessment and policy design in AI-driven web development. With quantified results indicating varying improvements based on guidance level—from unguided to expert prompts—the benchmark serves as a valuable tool for developers to make informed choices about model selection and implementation strategies in their accessibility efforts.