Researchers developed SIMAX, a framework designed to generate controlled clinical dialogue data with reference behavioral annotations to address the scarcity of scalable evaluation data for AI-driven communication coding systems. The system creates simulated clinician-patient interactions from predefined scenarios, personas, and voice conditions, utilizing specific codebooks to control overall communication quality and countable behaviors.
- SIMAX generated 3,388 simulated dialogues across three specialties, multiple visit stages, persona characteristics, and accent conditions.
- Automated assessments yielded mean UTMOS and WV-MOS scores of 3.03 and 2.61, with word error rates (WER) and character error rates (CER) of 0.07 and 0.05 respectively.
- Human evaluations resulted in a median MOS of 4.67 and a median clinical realism score of 3.00.
- Downstream evaluation demonstrated the framework's ability to assess how communication coding systems respond to behavioral targets and identify insufficient sensitivity in certain dimensions.
SIMAX provides a reproducible data foundation for developing, validating, and refining communication coding systems by offering controlled and annotated simulated dialogues.