DataGovBench benchmarks LLMs against real-world data complexities

Current benchmarks for evaluating Large Language Models (LLMs) in data analysis often fail to reflect real-world settings by focusing on fact retrieval from small tables while overlooking challenges like large multi-tabular datasets and exploratory insight discovery. To address this, the authors introduce DataGovBench, a benchmark derived from governmental open data designed to evaluate LLMs in practical scenarios.

• Table QA task requires solving complex decomposable questions and producing textual answers or visualizations.
• Table Insight task evaluates the ability of models to generate expert-level findings through exploratory data analysis.
• Comprehensive experiments with state-of-the-art LLMs, both with and without agentic frameworks, reveal significant performance gaps across both tasks.

These results suggest that current LLM-based systems remain far from satisfying the demands of real-world data analytics. DataGovBench provides a challenging benchmark for advancing research on LLMs capable of both answering analytical queries and discovering insights from data.