lab ARC Prize · 611d ago

ARC Prize 2024 winners published; SOTA on ARC-AGI-1 rises to 55.5%

from English

The ARC Prize organization has announced the winners of the 2024 competition and published the associated Technical Report, revealing that the state-of-the-art score on the ARC-AGI-1 benchmark increased from 33% to 55.5% during the event.

A total of 1,430 teams submitted 17,789 entries for the competition.
The Grand Prize remained unclaimed as no team solved the private evaluation set.
Progress on the secondary ARC-AGI-Pub leaderboard tracked closely with Kaggle scores despite access to larger frontier LLMs and significantly more compute.
All winning solutions and papers are open source and reproducible.

The competition is credited with catalyzing a shift in AI research toward new ideas beyond pure deep learning scaling, evidenced by recent releases from DeepMind and OpenAI that incorporate program synthesis or search.

Importance 1/3 ARC Prize Evaluation & benchmarks Research paper

Benchmarks

Benchmark	Model	Score
ARC-AGI 1	MindsAI	55.5%

Read original

arxiv arXiv cs.CL · 2d ago

M$^3$R-Bench introduces evidence-grounded benchmark for multimodal metaphor understanding

Researchers introduce M$^3$R-Bench, a unified benchmark containing 1,000 image-text instances with human-verified annotations designed to evaluate evidence-grounded multimodal metaphor understanding. The benchmark provides joint annotations for metaphor occurrence, Target-Source mapping, sentiment, and stage-wise explanations based on Conceptual Metaphor Theory.

arxiv arXiv cs.CL · 12d ago

Zing framework improves LLM social intelligence via SoMBench benchmark and Actio grounding

The report introduces Zing, an integrated framework designed to enhance the social intelligence of large language models by measuring, internalizing, and grounding social capabilities. The authors present SoMBench, a psychology-grounded benchmark spanning 17 secondary dimensions and 3,481 expert-verified instances, which reveals that current LLMs have significant room for improvement with no model reaching near-ceiling performance.

media Hugging Face Forums · 15d ago · 2 views

SRRM benchmark reveals Qwen2.5-1.5B outperforms 3B in long-context memory

A researcher is seeking an arXiv cs.CL endorsement for a paper introducing Simple Retrieval-Reconstruction Memory (SRRM), a lightweight benchmark designed to evaluate long-context memory in Small Language Models.

arxiv arXiv cs.CL · 24d ago

LLM fine-tuning with traits improves cross-rubric essay scoring

Researchers address the challenge of automated essay scoring when educators revise or introduce new rubrics, a scenario known as cross-rubric generalization. They propose a fine-tuning framework for Large Language Models that utilizes rubric-agnostic intermediate representations called "traits" alongside target-essay supervision during training.

arxiv arXiv cs.CL · 24d ago

Live Gurbani Tracking: Benchmark and Reference System for Sikh Kirtan Captioning

The authors present a benchmark and reference system for the live captioning of Sikh Kirtan, which involves the continuous sung recitation of verses from the Sri Guru Granth Sahib Ji. Unlike open-vocabulary transcription, this task requires exact word-for-word matches from the canonical scripture to avoid religiously inappropriate misspellings.