SHAP-Weighted Cross-Modal Expert Fusion for Emotion and Sentiment Recognition: Evidence and Limits

This paper analyzes XAI-guided adaptive fusion (XGAF) for multimodal emotion and sentiment recognition, focusing on how SHAP attribution reduction methods interact with expert feature dimensionalities. The authors demonstrate that sum-abs reduction preserves total attribution mass better than mean-abs or median-abs reductions, which can suppress high-dimensional cross-modal experts.

• On MELD 7-class emotion recognition, sum-abs XGAF achieves a weighted F1 of 0.5983 with Transformer aggregation, statistically matching early fusion (0.6018) and significantly outperforming late fusion (0.4598).
• On CMU-MOSEI 3-class sentiment recognition, sum-abs XGAF reaches 0.6519 weighted F1, slightly exceeding early fusion (0.6485) and late fusion (0.5696).
• Ablation studies indicate that performance gains stem primarily from adding cross-modal experts, particularly the trimodal expert, rather than complex per-sample routing.
• Diagnostics reveal that mean-abs and median-abs weights become nearly uniform, whereas sum-abs weights effectively concentrate on the trimodal expert.

The work provides a transparent empirical analysis of how SHAP reduction techniques, expert dimensionality, and cross-modal expert design influence modular multimodal fusion performance.