Show HN: I benchmarked how good LLMs are at proofreading English (github.com)

A new benchmarking tool called ErrataBench has been introduced to evaluate the proofreading capabilities of large language models (LLMs). This tool utilizes a straightforward agent loop to enable models to identify and rectify errors in various text samples, covering categories such as spelling, grammar, word choice, and typos. The benchmark features approximately five errors per 1,000 words and has been tested on a diverse set of 51 model variants across over 1,600 samples. Users can access and view results online at revise.io/errata-bench, making it a valuable resource for observing LLM performance in this domain. The significance of this benchmark lies in its potential to advance the AI/ML community's understanding of LLM capabilities in real-world applications, particularly those related to text editing and proofreading. By providing clear scoring metrics and methods for comparing different models, ErrataBench allows researchers and developers to assess not only accuracy but also the efficiency and cost-effectiveness of various models. The benchmark is compatible with any system supporting an OpenAI-compatible API, which broadens its accessibility and applicability for both released and unreleased models, encouraging experimentation and innovation in error correction technologies.