Multimodal Voice Activity Projection framework improves social robot turn-taking

The authors present a Multimodal Voice Activity Projection (MM-VAP) framework that extends audio-only voice activity prediction to synchronized audio-visual inputs for social robots in mediator settings. The approach adapts pretrained audio-visual backbones via Low-Rank Adaptation and uses inter-speaker attention to model relational dynamics for projecting future voice activity.

• Introduces a semantic consistency loss to regularize the 256-state output space based on higher-level dialogue patterns.
• Demonstrates improvements over current baselines on the NoXi and NoXi+J datasets, particularly for specific turn-taking events.
• Validates suitability for mediation-oriented human-robot interaction through additional evaluation on the Haru EDR corpus.

This method enables robots to anticipate conversational dynamics rather than merely reacting to pauses, enhancing their effectiveness in human-human interaction scenarios.