Evaluating OCR-to-Markdown systems is fundamentally broken (nanonets.com)

Recent discussions in the AI/ML community highlight the fundamental flaws in evaluating OCR-to-Markdown systems, which convert PDFs and document images into Markdown format. Unlike traditional OCR, this process is complex, as it requires restoring not just textual content but also layout, reading order, and representation choices. Current evaluation benchmarks rely heavily on string matching and heuristic methods, which often misclassify valid outputs as incorrect due to their inherent lack of flexibility in assessing varied valid outputs. The significance of identifying these evaluation issues cannot be understated; inaccurate benchmarks can hinder the development and deployment of reliable OCR systems. As a solution, the use of large language models (LLMs) for evaluation has gained traction, offering a more nuanced understanding of semantic equivalence by recognizing different valid representations of content. While LLM-based evaluation presents challenges such as non-determinism and sensitivity to prompt design, its ability to assess correctness from a contextual standpoint makes it a promising alternative. This shift underscores a need for the AI/ML community to re-evaluate how OCR systems are assessed, prioritizing semantic accuracy over rigid formatting criteria.