Benchmark · reasoning

DROP

saturated 0 results 0 models

DROP (Discrete Reasoning Over Paragraphs) is a reading-comprehension benchmark of about 96K crowdsourced questions over Wikipedia passages that require discrete reasoning over the text — arithmetic, counting, sorting, and comparison — scored by Exact Match (EM) and a numeracy-aware macro-averaged F1.

Example: A short passage (often an NFL game recap or a history article) paired with a question that can't be answered by copying a single span — for instance, subtracting two quantities stated in the text, counting how many events of a given type occurred, or ordering entities to decide which came first or last.
Scoring: Two metrics, macro-averaged over all questions: Exact Match (the normalized prediction equals the gold answer) and F1 (bag-of-words token-overlap F1 between prediction and gold, with multi-span answers optimally aligned span to span). It is numeracy-aware: if a number in the answer does not exactly match the gold number, that F1 is 0. A gold answer may be a number, one or more text spans, or a date.
Verification: Predictions are scored against the gold answer set by the official DROP evaluation script after normalization (lowercasing; removing articles, punctuation, and extra whitespace), taking the best score over all acceptable gold answers. Public leaderboards report EM and F1 on a hidden test split.
Why it matters: Unlike span-extraction datasets, DROP requires composing several facts with symbolic operations (add, subtract, count, compare), so it reveals whether a model truly reasons numerically over text instead of pattern-matching a single answer span. It was built adversarially — keeping questions a strong baseline reader got wrong — and remains a standard probe of discrete and numerical reasoning.

Worked example

Task

Passage: 'The Broncos opened the scoring with a 24-yard field goal; the Chargers replied with a 5-yard rushing touchdown; before halftime the Broncos added a 43-yard field goal and the Chargers threw a 12-yard touchdown pass.' Question: 'How many yards longer was the Broncos' longer field goal than their shorter field goal?'

Solution

Identify the Broncos' two field goals: 24 and 43 yards. Subtract the shorter from the longer: 43 − 24 = 19. Final answer: 19 (number).

Walkthrough

The answer is not a span in the passage; it must be derived by selecting the two Broncos field goals and subtracting — exactly the discrete arithmetic DROP targets. Grading: the predicted number must exactly match the gold answer 19, yielding EM = 1 and F1 = 1 (the numeracy check passes).

No verified scores reported yet for this benchmark.