LLM4SDM study finds Gemma3:12b best at assessing shared decision making in Dutch melanoma consultations

The LLM4SDM study evaluates open-source smaller language models (OS-sLLMs) for automated assessment of shared decision making using the Observer OPTION12 framework. Focusing on privacy-preserving locally deployable models and Dutch melanoma consultation transcripts, the research compares general-domain and medical-domain OS-sLLMs against expert annotations.

• Gemma3:12b achieves the strongest agreement with human annotations, recording a Pearson r of 0.51 and Spearman rho of 0.59.
• General-domain models outperform medical-domain models, which exhibit substantial hallucination and instruction-following failures.
• Item-level and qualitative analyses reveal systematic challenges related to temporal discourse reasoning, conversational role attribution, and evidence grounding.
• The study introduces a Judge-LLM consensus framework designed to support disagreement resolution among multiple models.

While current OS-sLLMs cannot replace human annotators, they offer a promising foundation for privacy-preserving human-in-the-loop SDM assessment.