The authors introduce LongMedBench, a real-world electronic health record (EHR) benchmark designed to evaluate long-horizon clinical decision-making in medical agents. Unlike prior evaluations that focus on short-context knowledge QA, this benchmark addresses the longitudinal nature of medical care by aggregating evidence across repeated visits and evolving treatments.
- Constructed via a pipeline integrating MIMIC-IV admission records and clinical notes into time-series event streams and long-context memory datasets.
- Comprises 335 patients with an average of 19.72 inpatient visits per patient and 44.91 medical events per visit.
- Features an evaluation taxonomy with three suites: fact-based QA, temporal reasoning, and long-horizon decision-making.
- Experiments reveal that while recent LLMs handle explicit timestamps well, they struggle with implicit time inference.
- RAG and agent memory systems improve information retrieval performance, but decision-making performance remains highly dependent on the model's immediate context.
This benchmark enables a more realistic assessment of how agents understand and leverage historical patient information over extended horizons.