Researchers introduce MedRealMM, a large-scale benchmark for multimodal online medical consultation constructed from de-identified patient-doctor interactions collected from a nationwide Chinese internet hospital. The dataset utilizes a Multimodal Clinical Challenge Point extraction framework to convert authentic consultation trajectories into standardized next-response generation tasks paired with physician-refined rubrics.
- The current release contains 5,620 real-world multimodal cases spanning 64 clinical departments.
- Evaluation of 19 general-purpose and medical-specialized LLMs reveals that image information is critical for reliable clinical performance.
- Current frontier models remain below online physician response levels, with safety-sensitive error avoidance identified as a central bottleneck.
MedRealMM provides a realistic and reproducible benchmark for evaluating multimodal medical reasoning in real-world settings, addressing the limitations of existing benchmarks that rely on synthetic conversations or poor evaluation metrics.