CSB dataset reveals MLLMs eliminate gap in complex social behavior scene description accuracy

This study introduces the Complex Social Behavior (CSB) dataset, containing 100 images of complex social interactions, to evaluate vision-language models over a decade (2017-2025). The research analyzes five visual-cognitive error types across four pre-Multimodal Large Language Models (MLLMs) and five MLLMs.

• Pre-MLLMs achieved lower accuracy than bottom-ranked human descriptions, while MLLMs matched top-ranked humans.
• MLLMs eliminated the accuracy gap between simple MS-COCO scenes and complex CSB scenes.
• Detection, recognition, and hallucination errors were found to have the highest impact on scene description accuracy.
• Spatial dependence errors remain the only significant error type for MLLMs in this evaluation.

The findings provide a more thorough evaluation of how visual language models have advanced, showing that MLLMs have almost eliminated most error types except for spatial dependence.