media Hugging Face Forums · 2h ago · open_models

SRRM benchmark reveals Qwen2.5-1.5B outperforms 3B in long-context memory

from English

A researcher is seeking an arXiv cs.CL endorsement for a paper introducing Simple Retrieval-Reconstruction Memory (SRRM), a lightweight benchmark designed to evaluate long-context memory in Small Language Models.

Unlike existing retrieval-oriented benchmarks, SRRM jointly evaluates targeted retrieval, global information reconstruction, and a new metric called Memory Preservation Ratio (MPR). The authors tested the Qwen2.5 model family (0.5B, 1.5B, and 3B) under context loads of 10–100 factual statements.

The results showed that the Qwen2.5-1.5B model consistently outperformed the larger 3B model across retrieval accuracy, reconstruction performance, and MPR, suggesting that parameter count alone does not necessarily improve long-context memory preservation.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Research paper

Read original

arxiv arXiv cs.CL · 9d ago

LLM fine-tuning with traits improves cross-rubric essay scoring

Researchers address the challenge of automated essay scoring when educators revise or introduce new rubrics, a scenario known as cross-rubric generalization. They propose a fine-tuning framework for Large Language Models that utilizes rubric-agnostic intermediate representations called "traits" alongside target-essay supervision during training.

arxiv arXiv cs.CL · 9d ago

Live Gurbani Tracking: Benchmark and Reference System for Sikh Kirtan Captioning

The authors present a benchmark and reference system for the live captioning of Sikh Kirtan, which involves the continuous sung recitation of verses from the Sri Guru Granth Sahib Ji. Unlike open-vocabulary transcription, this task requires exact word-for-word matches from the canonical scripture to avoid religiously inappropriate misspellings.

arxiv arXiv cs.CL · 9d ago

Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

The article highlights a structural failure in building synthetic corpora for LLM-as-judge bias studies, where shared decoding-budget parameters can truncate generated hallucinated answers. This truncation caused a 32-point cross-lingual collapse in judge selection accuracy, which was only detectable through manual inspection of raw generations rather than aggregate statistical checks.

arxiv arXiv cs.CL · 9d ago

CAVE-ABSA framework generates validated aspect-level counterfactuals

The authors propose CAVE-ABSA, a Constraint-Aware Validated Editing framework designed to generate and validate aspect-level counterfactuals for Aspect-Based Sentiment Analysis (ABSA). This approach addresses the challenge of flipping sentiment for a specific target aspect while preserving the meaning and sentiment of non-target aspects.

arxiv arXiv cs.LG · 9d ago

Multi-Expert Routing for Multi-Domain Low-Resource OCR: A Manchu Case Study

The article presents a multi-expert system for historical Manchu OCR that handles visually distinct writing styles like regular script, running script, and semi-cursive chancery hand despite limited labeled data. The approach reuses checkpoints from an iterative fine-tuning process as domain specialists and employs a lightweight page-level image classifier to dispatch pages based on visual style.