Why Tables Are the Hardest Problem in Document AI (www.runpulse.com)

Tables, not paragraphs, are the hardest part of document AI because they are geometric, structured objects whose meaning depends on spatial relationships that OCR and language models routinely destroy. Modern pipelines break pages into tokens or image patches (e.g., ViT-style 16×16 patches), flatten 2D grids into 1D sequences, and lose row/column coordinates and merged spans. The worst failures are subtle: characters and numbers are extracted correctly but misaligned with headers or other cells, so a P&L or rent roll looks “right” at surface level while its semantics and aggregations are wrong—errors that propagate silently through downstream models and decisions. Fixing this requires treating tables as first-class mathematical objects, not text boxes. Practical systems need granular cell-level bounding boxes, header-stack reconstruction so each cell inherits hierarchical semantics, cross-page stitching for multi-page tables, and constraint-validation layers that enforce arithmetic and subtotal consistency. Outputs must be deterministic and auditable. Most benchmarks (FUNSD, DocVQA) use toy tables and understate real-world complexity—nested headers, rotated text, footnotes, inconsistent units—so high benchmark scores don’t translate to production readiness. The path forward is geometry-first document AI: reconstruct grids and spans before semantics, anchor every value to coordinates, and validate structure with mathematical constraints.