media Hugging Face Forums · 1h ago · open_models

Author proposes open k registry for evidence saturation in local LLMs

from English

The author has published a pilot paper and an open registry on Hugging Face to measure the evidence-saturation point (k*) of language models. This metric determines the optimal number of evidence fragments to inject into prompts, as adding more context does not monotonically improve correctness and can increase cost or epistemic contamination.

The study finds that the correctness-optimal k ranges from 1 to the full evidence set depending on task type, suggesting a single context profile is insufficient.
An open Evidence-k Registry has been created on Hugging Face, seeded with 35 task-level measurements from the paper.
The registry invites contributions from local-model operators to document model revisions, quantizations, served backends, and reliability metrics.

This initiative aims to provide a reproducible protocol for calibrating context windows per model and backend to balance correctness with efficiency.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Retrieval & RAG

Read original

arxiv arXiv cs.CL · 6d ago

LLM-as-a-judge signals fail to optimize closed-loop table recognition

A study using FinTabNet and OmniDocBench demonstrates that LLM-as-a-judge signals are insufficient for optimizing closed-loop regeneration in table recognition. The research reveals that judge scores frequently tied, rankings were not reproducible, and the system failed to recover better candidates produced by iteration.

arxiv arXiv cs.CL · 6d ago

DeepStress stress-tests deep search agents against unreliable evidence

The authors introduce DeepStress, a framework for stress-testing deep search agents by replacing their retrieval modules with a controlled synthetic environment. This approach allows researchers to manipulate the frequency and quality of evidence across three dimensions: trustworthiness, relevance, and factuality.

arxiv arXiv cs.CL · 6d ago · 2 views

Hindcast closes data leakage in LLM forecaster evaluation

The Hindcast framework evaluates LLM forecasters by replaying resolved Polymarket prediction markets against a frozen snapshot of public Reddit, grading models as if they existed at a specific past date. This approach prevents two common channels of answer leakage: retrieval of post-event reports and training data contamination from events that occurred after the model's release.

arxiv arXiv cs.CL · 7d ago

RCWT measures task-budget displacement from coordination content in LLM calls

The Roundtable Context Window Test (RCWT) is a controlled protocol designed to measure the task-budget displacement effect caused by placing coordination content within the finite prompt of multi-agent and memory-augmented LLM systems.

arxiv arXiv cs.CL · 7d ago

LakeQuest: A Three-Domain Benchmark for Grounded Question Answering across Data Lakes

Researchers introduce LakeQuest, a human-validated benchmark of 9,846 question-answer pairs designed to evaluate end-to-end retrieve-and-synthesize pipelines over realistic data lakes. The dataset spans three diverse domains—AI/ML metadata, retail banking, and multimodal biomedical drug information—and pairs every question with exact, modality-aware evidence pointers.