Researchers introduce LakeQuest, a human-validated benchmark of 9,846 question-answer pairs designed to evaluate end-to-end retrieve-and-synthesize pipelines over realistic data lakes. The dataset spans three diverse domains—AI/ML metadata, retail banking, and multimodal biomedical drug information—and pairs every question with exact, modality-aware evidence pointers.
- LakeQuest isolates source discovery from cross-modal synthesis to expose critical failure modes in modern QA systems.
- Baseline evaluations of standard Retrieval-Augmented Generation (RAG) and agentic tool-use methods reveal that high-quality retrieval does not guarantee correct reasoning.
- Systems consistently struggle with relation chaining in metadata graphs, policy grounding in bank ledgers, and joint tabular QA in biomedical contexts.
The benchmark highlights the need for robust discovery and faithful cross-file composition mechanisms in future agentic QA systems.