arxiv arXiv cs.AI · 46d ago · src: 49d ago · research

BabelJudge: Measuring LLM-as-a-Judge Reliability Across Languages and Agent Trajectories

from English

BabelJudge introduces an open-source framework to measure four key bias modes in LLM judges across languages and agent trajectories. It reveals a significant reliability drop from Hindi to Swahili—0.714 to 0's 0.550—highlighting cross-lingual degradation invisible to raw accuracy. The framework enables bias-aware evaluation without human labels, using controlled perturbations to create known gold labels, and extends to agentic workflows with new metrics on tool accuracy and hallucination detection.

Importance 1/3 New harness with differentiators arXiv cs.AI Hugging Face Evaluation & benchmarks Retrieval & RAG

Read original

arxiv arXiv cs.CL · 51d ago · 1 view

Tool-Intent Stabilization in Streaming RAG

A study measures tool-intent stabilization in Streaming RAG, defining when speculative tool queries converge to correct answers. On the CRAG benchmark, 73.9% of queries allow substantial latency hiding, with early stabilization observed in questions with verbatim retrievable evidence. Question type significantly predicts early versus late stabilization, informing when speculative triggers are effective.

media Hugging Face Forums · 12d ago

Calibra: Robot Dataset Observability for LeRobot

Calibra is an open-source toolkit designed to audit robot datasets and identify quality issues before training. It provides tools for building quality-aware coresets and estimating training outcomes.

media Hugging Face Forums · 18d ago · 1 view

Author proposes open k registry for evidence saturation in local LLMs

The author has published a pilot paper and an open registry on Hugging Face to measure the evidence-saturation point (k*) of language models. This metric determines the optimal number of evidence fragments to inject into prompts, as adding more context does not monotonically improve correctness and can increase cost or epistemic contamination.

arxiv arXiv cs.AI · 23d ago · 1 view

RAGU introduces multi-step GraphRAG engine with compact Meno-Lite-0.1 extractor

RAGU is an open-source modular GraphRAG engine that improves structured knowledge integration by separating entity extraction from consolidation. It utilizes a two-stage typed extraction process, DBSCAN-backed deduplication, LLM summarization, and Leiden community detection to reduce noise and brittleness found in single-pass systems.

arxiv arXiv cs.CL · 24d ago

LLM-as-a-judge signals fail to optimize closed-loop table recognition

A study using FinTabNet and OmniDocBench demonstrates that LLM-as-a-judge signals are insufficient for optimizing closed-loop regeneration in table recognition. The research reveals that judge scores frequently tied, rankings were not reproducible, and the system failed to recover better candidates produced by iteration.