Researchers introduce ThReadMed-QA, a multi-turn medical dialogue dataset of 2,437 conversation threads derived from AskDocs, to evaluate whether large language models can reliably detect and correct patient misconceptions over time. The study highlights that current evaluation frameworks fail to capture how false beliefs persist or evolve across multiple interaction turns.
- ThReadMed-QA comprises 8,204 question-answer pairs designed for systematic evaluation of misconception correction in multi-turn contexts.
- Five LLMs were evaluated using a rubric-based LLM-as-a-Judge framework scoring their ability to identify and correct false beliefs.
- GPT-5 and Claude-Haiku corrected misconceptions around 85% on initial questions but dropped to roughly 50% within two follow-ups.
- An oracle analysis replacing prior model outputs with physician responses showed that error propagation drives much of the performance degradation.
The findings indicate that even frontier models degrade substantially over subsequent turns, leading to inconsistent and potentially unsafe guidance in patient-facing settings.