How OCR Models Work (harsehaj.substack.com)

Modern OCR models are moving from multi-stage, hand-crafted pipelines to end-to-end, encoder–decoder systems that literally learn to “read” images. A vision encoder (ViT or CNN backbone) ingests raw RGB pixels and converts them into a semantic map — learned features that implicitly handle glare, skew, segmentation and denoising. State-of-the-art systems like “deepseek-ocr” compress millions of pixels into a few dozen visual tokens (≈10× compression), each encoding regions such as “upper-left paragraph” or “table cell.” A transformer-based decoder (a language model) then autoregressively generates characters or subwords while attending back to those visual tokens, allowing each output step to consult the visual context that produced it. This architecture matters because it replaces brittle, rule-based preprocessing with learned behaviors, enabling robustness to new fonts, layouts and handwriting and supporting structured outputs (JSON, key–value pairs, entities) natively. Cross-attention between generated text and visual tokens enables self-healing corrections and context-aware disambiguation, while end-to-end training optimizes every implicit operation to minimize final loss — often with fewer tokens and lower compute than older systems. The practical implications include better document understanding, easier deployment for assistive tech and KVP extraction, and the broader idea that “text as images” could become a unified input pathway for multimodal reading tasks.