Benchmark · multimodal

Video-MME

0 results 0 models

Video-MME is a comprehensive benchmark that evaluates multimodal LLMs on video understanding across short, medium, and long videos. Performance is reported as multiple-choice accuracy over 2,700 human-annotated questions on 900 videos (254 hours) spanning six visual domains.

Example: A model watches a video clip (optionally with its subtitle and audio tracks) and answers a four-option multiple-choice question that requires reasoning over visual content across time — for example, identifying the order in which events happen in a medium-length sports clip.
Scoring: The metric is accuracy: the percentage of four-option multiple-choice questions answered correctly. Scores are usually reported both without and with subtitles, broken down by video length (short/medium/long) and domain, then averaged into an overall number.
Verification: The model's chosen letter (A/B/C/D) is parsed from its output and compared by exact match to the expert-annotated gold answer; the fraction correct is the accuracy. All videos and questions are manually collected and annotated to minimize data leakage, so numbers are comparable on the public leaderboard.
Why it matters: It was the first comprehensive benchmark to test multimodal LLMs on real videos spanning a wide range of durations (up to about an hour) and diverse domains, using multiple modalities (frames, subtitles, audio) — revealing genuine temporal and long-video reasoning rather than single-frame recognition.

Worked example

Task

A medium-length (~10 min) clip from the Knowledge domain is provided together with its subtitle track. Question: 'What does the presenter give as the main reason the described technique became widely adopted?' Options: A) It was cheaper than the alternatives; B) It required no special training; C) It produced more consistent results; D) It was legally mandated. (Illustrative item in the benchmark's format.)

Solution

Combine the on-screen visuals with the subtitle/audio narration, rule out the three options the clip does not support, and pick the one it does. Final answer: C.

Walkthrough

Video-MME questions are single-correct, four-way multiple choice, so exactly one option (C) matches the clip's evidence; grading is exact letter match against the expert gold label, scored as accuracy (correct ÷ total).

No verified scores reported yet for this benchmark.