Researchers analyzed 2,053 real patient-chatbot conversations to demonstrate that communication patterns and emotional expression vary widely among users, challenging the reliance on idealized simulated patients in AI development. They developed a simulator modeling clinical content, emotion, strategy, and style, which produced conversations nearly indistinguishable from real ones in a Turing-inspired evaluation.
- The team used five distinct patient personae across 1,64 clinician-graded cases to evaluate four LLMs in urgency assessment.
- Analysis revealed that communication style significantly alters triage outcomes.
- Human graders achieved only 55% accuracy in distinguishing simulated from real conversations.
The authors argue that patient-centered conversational AI must accommodate communication diversity, as systems designed for idealized interactions risk underperforming and amplifying health disparities in real-world deployment.