Evaluating OCR on the Community Memory Corpus (ztoz.blog)

A recent evaluation of optical character recognition (OCR) models on the Community Memory corpus—5,500 scanned posts from Berkeley's early social network—highlights the competitive capabilities of general-purpose language models against specialized OCR engines. The study tested five models: Document AI, Gemini 3.7 and 3.8, Mistral OCR, and Qwen3, aiming to prepare the corpus for a potential digitization project. Notably, results showed that modern LLMs outperformed traditional OCR models, with Gemini 3.7 emerging as the best performer in character and word error metrics, while Gemini 3.8 regressed in performance compared to its predecessor. This finding is significant for the AI/ML community as it underscores the growing efficacy of LLMs in tasks traditionally dominated by specialized OCR solutions. Evaluating models based on character error rates (CER) and word error rates (WER), the research concluded that tasks using OCR results would need to tolerate significant errors—up to four character errors per line. This research opens pathways for leveraging advanced LLMs in more complex OCR applications while raising questions about the robustness and practical implementation of these technologies in digitization efforts, especially when addressing historical texts with potential inaccuracies.