Know Your Source: A Public Knowledge Store for Media Background Checks

The authors introduce MEDIAREF, a publicly available knowledge store of web-sourced documents designed to enable reproducible and low-cost evaluation of media background checks (MBCs). This tool addresses the reliance on costly proprietary search APIs in recent source-critical reasoning approaches.

• MEDIAREF supports the evaluation of MBC generation across 200 media sources.
• The study describes a reproducible methodology for constructing and updating the collection.
• Widely used LLMs are assessed on the MBC generation task using this new resource.
• Automatic and qualitative evaluations demonstrate that MEDIAREF supports higher-quality MBC generation.

MEDIAREF mitigates the limitations of existing approaches by providing an open alternative for assessing evidence source credibility in automated fact-checking systems.