Multi-Modal vs. Text-Based: Benchmarking LLM Strategies for Invoice Processing (arxiv.org)

Researchers benchmarked eight multi-modal large language models across three families (GPT-5, Gemini 2.5, and open-source Gemma 3) on three publicly available invoice datasets to compare two zero-shot processing strategies: (1) native image-based understanding using each model’s vision-language capabilities and (2) a structured pipeline that converts invoice pages into markdown (text-first parsing) before prompting. The study evaluates extraction and parsing quality without any task-specific fine-tuning, providing a direct comparison of out-of-the-box performance for real-world invoice layouts that include tabular data, line items, and heterogeneous formatting. Code and data are available for reproducibility. Key finding: native image processing generally outperforms the markdown/text-first approach, indicating modern multi-modal LLMs are increasingly effective at handling layout, spatial cues, and visual context in invoices. However, performance varies by model family and by document characteristics (e.g., dense tables, noisy scans), so a text-based pipeline can still be preferable when high-quality OCR is available or when deterministic, structured outputs are required. For practitioners this suggests fewer end-to-end OCR+NLP pipelines may be necessary for many invoice tasks, but trade-offs remain around consistency, latency, cost, and edge-case robustness—making these benchmarks a practical guide for selecting models and strategies in automated document systems.