Benchmarking LLMs/VLMs on document parsing, extraction, VQA (nanonets.com)

A new benchmarking initiative, the Intelligent Document Processing (IDP) Leaderboard, has been launched to assess the capabilities of different AI models in document parsing and extraction tasks. With three open benchmarks and over 16 models evaluated against more than 9,000 real documents, the leaderboard focuses on critical tasks such as OCR, table extraction, key information extraction, visual question answering, and long document understanding. Unlike typical benchmarks that provide a single score, this platform allows users to delve into model specifics, enabling them to compare performance and relevance for their unique use cases. Significantly, the results reveal that less costly models can match or even outperform more expensive counterparts in certain tasks, such as extraction. For instance, Nanonets OCR2+ offers high accuracy at a fraction of the cost of other models, while more advanced models like Gemini 3.1 Pro excel in complex reasoning and visual question answering tasks. The initiative aims to democratize model selection in the AI/ML community by providing transparent access to performance data and encouraging users to make informed decisions based on their individual document processing needs. Additional models and datasets are expected to be integrated into the leaderboard over time, ensuring ongoing relevance and adaptability.