Benchmark · multimodal

LongVideoBench

0 results 0 models

LongVideoBench is a multiple-choice question-answering benchmark that tests how well multimodal models understand long, subtitled videos (up to one hour) presented as interleaved frames and text. Performance is measured by accuracy — the percentage of questions for which the model picks the correct option.

Example: A 'referring reasoning' question over a long video: the prompt includes a referring query that points to a specific moment (e.g., 'the scene right after the presenter opens the red box'), and the model must retrieve that moment from the sampled frames and subtitles and choose the correct option about what happens there.
Scoring: The metric is multiple-choice accuracy: score = (questions answered correctly) / (total questions). A question counts as correct only when the model's chosen option matches the single gold option. Results are typically also broken down by the four duration groups (8–15s, 15–60s, 3–10min, 15–60min) and by question category.
Verification: An answer is accepted when the predicted option label matches the gold label exactly. The validation split (1,337 questions) ships with public answers, so accuracy can be computed locally; test-split answers are held out and scored via the official leaderboard, so publicly reported scores use the validation split.
Why it matters: Most video benchmarks can be solved from a handful of frames; LongVideoBench's referring-reasoning design forces models to locate and reason over specific details buried in hour-long interleaved video-plus-subtitle inputs, exposing how accuracy degrades as videos get longer and testing genuine long-context multimodal retrieval.

Worked example

Task

Representative item. Input: sampled frames + timestamped subtitles from a ~25-minute cooking vlog. Referring-reasoning question: Referring query: the moment when the host first adds salt to the pan. Question: which ingredient does the host add immediately after that moment? A) Garlic B) Sugar C) Chopped onions D) Olive oil E) Black pepper

Solution

Locate the referred moment (the first time salt is added) among the sampled frames using subtitle/image cues; inspect the frames immediately after it; identify the next ingredient added. Correct option: C) Chopped onions.

Walkthrough

The referring query pins a single anchor moment, and only one option matches what the following frames actually show; the distractors are plausible ingredients that appear elsewhere or not at all, so guessing without retrieving the right segment fails. Grading: exact match of the chosen letter against the gold option — the item counts toward overall accuracy.

No verified scores reported yet for this benchmark.