A study analyzing 2,053 real patient-chatbot conversations reveals that communication patterns and emotional expression vary widely among users, challenging the reliance on idealized simulated patients in AI development. The researchers developed a simulator modeling clinical content, emotion, strategy, and style, which produced conversations nearly indistinguishable from real ones in a Turing-inspired evaluation.
- Analysis of 2,053 real conversations showed significant variation in communication patterns and emotional expression across users.
- A new patient simulator models clinical content, emotional state, conversational strategy, and communication style separately.
- In a Turing-inspired evaluation with 15 human graders, simulated conversations achieved a classification accuracy of only 55%.
- Testing four LLMs on 1,164 clinician-graded cases using five distinct patient personae found that communication style significantly alters triage outcomes.
The authors argue that patient-centered conversational AI must accommodate communication diversity to avoid underperforming and amplifying health disparities when deployed in the real world.