arxiv arXiv cs.CL · 3h ago · research

M$^3$R-Bench introduces evidence-grounded benchmark for multimodal metaphor understanding

from English

Researchers introduce M$^3$R-Bench, a unified benchmark containing 1,000 image-text instances with human-verified annotations designed to evaluate evidence-grounded multimodal metaphor understanding. The benchmark provides joint annotations for metaphor occurrence, Target-Source mapping, sentiment, and stage-wise explanations based on Conceptual Metaphor Theory.

M$^3$R-Bench evaluates models on cross-modal reasoning by requiring evidence identification, mapping establishment, and sentiment inference.
Evaluations reveal that existing models often overlook visual evidence and rely on superficial textual cues, exposing a cross-modal evidence-mapping mismatch.
To address this, the authors propose M$^3$R-Reasoner, which combines curriculum-based reasoning supervision with task-aware reinforcement learning.
The 8B-parameter M$^3$R-Reasoner outperforms larger proprietary MLLMs and improves Visual Evidence and Sentiment Justification scores over GPT-5.5 by 28.45 and 30.11 points respectively.

The benchmark highlights the difficulty of assessing whether models establish mappings grounded in visual and textual cues, while the proposed reasoner demonstrates improved alignment with metaphor interpretation.

Importance 2/3 Beats a top-lab benchmark arXiv cs.CL Evaluation & benchmarks Research paper

Related articles

arxiv arXiv cs.CL · 10d ago

Zing framework improves LLM social intelligence via SoMBench benchmark and Actio grounding

The report introduces Zing, an integrated framework designed to enhance the social intelligence of large language models by measuring, internalizing, and grounding social capabilities. The authors present SoMBench, a psychology-grounded benchmark spanning 17 secondary dimensions and 3,481 expert-verified instances, which reveals that current LLMs have significant room for improvement with no model reaching near-ceiling performance.

media Hugging Face Forums · 13d ago · 1 view

SRRM benchmark reveals Qwen2.5-1.5B outperforms 3B in long-context memory

A researcher is seeking an arXiv cs.CL endorsement for a paper introducing Simple Retrieval-Reconstruction Memory (SRRM), a lightweight benchmark designed to evaluate long-context memory in Small Language Models.

arxiv arXiv cs.CL · 22d ago

LLM fine-tuning with traits improves cross-rubric essay scoring

Researchers address the challenge of automated essay scoring when educators revise or introduce new rubrics, a scenario known as cross-rubric generalization. They propose a fine-tuning framework for Large Language Models that utilizes rubric-agnostic intermediate representations called "traits" alongside target-essay supervision during training.

arxiv arXiv cs.CL · 22d ago

Live Gurbani Tracking: Benchmark and Reference System for Sikh Kirtan Captioning

The authors present a benchmark and reference system for the live captioning of Sikh Kirtan, which involves the continuous sung recitation of verses from the Sri Guru Granth Sahib Ji. Unlike open-vocabulary transcription, this task requires exact word-for-word matches from the canonical scripture to avoid religiously inappropriate misspellings.

arxiv arXiv cs.CL · 22d ago

Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

The article highlights a structural failure in building synthetic corpora for LLM-as-judge bias studies, where shared decoding-budget parameters can truncate generated hallucinated answers. This truncation caused a 32-point cross-lingual collapse in judge selection accuracy, which was only detectable through manual inspection of raw generations rather than aggregate statistical checks.