CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs(2024) (arxiv.org)

Researchers have introduced CharXiv, a new evaluation suite designed to identify shortcomings in the chart understanding capabilities of Multimodal Large Language Models (MLLMs). Traditional benchmarks have relied on simplified, template-based charts, which exaggerate model effectiveness. CharXiv addresses this issue by featuring 2,323 complex and diverse charts sourced from arXiv papers, encompassing both descriptive and reasoning questions that test a model's ability to interpret intricate visual data. Initial findings reveal a stark discrepancy between model performance and human proficiency, with top models like GPT-4o achieving 47.1% accuracy compared to 80.5% for humans. This initiative is significant for the AI/ML community as it highlights the critical need for better evaluation metrics that reflect real-world tasks, particularly in fields where accurate data interpretation is essential, such as science and finance. By establishing a more realistic framework for assessing chart comprehension, CharXiv is poised to drive future research and improvements in MLLM capabilities, ultimately leading to better tools and applications in data-driven environments.