Study finds MCoT improves but fails to overcome LLM bias in CBT-guided affective reasoning

Researchers investigated why large language models (LLMs) struggle with Cognitive Behavioral Therapy (CBT) by introducing a knowledge-guided framework that decomposes user narratives into Beck's Cognitive Conceptualization structure and uses Natural Language Inference for SNOMED CT concepts. The study evaluated this approach across three open-weight LLMs and 14 RealCBT-derived case studies using a new metric called Protocol Leverage Force (F) to measure behavioral shifts away from default responses.

• The framework employs a Multiple Chain-of-Thought (MCoT) strategy selection between Validation & Reflection, Socratic Questioning, or Alternative Perspectives.
• Single chain-of-thought prompting failed to change LLM behavior, while MCoT guided strategy selection more effectively.
• Despite the improvement, the effect remained within 1% (approx. 1.2-1.3%), with all models remaining biased toward Validation & Reflection.

The results demonstrate that CBT knowledge alone does not ensure effective application, providing the affective-computing community with instrumentation to measure where LLMs fall short.