The authors introduce DeepStress, a framework for stress-testing deep search agents by replacing their retrieval modules with a controlled synthetic environment. This approach allows researchers to manipulate the frequency and quality of evidence across three dimensions: trustworthiness, relevance, and factuality.
- The framework replaces standard retrieval with a synthetic environment to control document reliability.
- Testing on HotpotQA and BrowseCompPlus reveals substantial differences in how agents handle unreliable information.
- The study proposes new metrics to better document system outcomes and interactions between parametric and retrieved knowledge.
This work addresses the under-explored robustness of search agents to poor-quality evidence, which can lead to dramatic failures in real-life applications despite performing well on realistic benchmarks.