Researchers introduce a benchmark to evaluate how LLM agents adapt when the reliability of available tools changes silently during an ongoing session. The framework mounts tool-skill libraries with redundancies and uses a branched schedule to shift the reliable tool group at hidden boundaries.
- Agents default to settling on a small recurring routine within a few turns of each boundary, concentrating call shares on discrete values.
- Set-shifting accuracy is scored as the joint probability of routing to the target tool group in every post-shift window.
- Testing open-weight LLMs in an open-source agentic harness reveals qualitatively distinct failure modes across the same routines.
- The study finds that set framing, presenting tools as competing or complementary, shifts routing dynamics.
This evaluation highlights specific failure modes and the impact of tool presentation on agent behavior when facing hidden reliability shifts.