media Hugging Face Forums · just now · open_models

SplatRAG 3+ achieves SciFact R@10 0.8804 with slang in memory

from English

SplatRAG 3+ demonstrates that a local Gaussian-splat memory can maintain high retrieval accuracy on the SciFact benchmark while simultaneously storing 5,000 Urban Dictionary slang rows without degradation.

The system uses an append-only cold log with BM25 and a 64-d HNSW index, where the 3-d field serves as a PCA visualization of the vectors rather than the index itself.
SciFact evaluation on 300 claims yielded R@10 of 0.8804 and nDCG@10 of 0.7624, surpassing previous floor metrics of 0.88 and 0.75.
Performance improvements were driven by Reciprocal Rank Fusion (RRF) combining BM25 with the full 4096-d Qwen3-Embedding-8B cosine output, utilizing previously unused dimensions.
Evaluation filters ensure domain-scifact results are not contaminated by slang entries, confirming the science documents remain distinct within the shared field.

The results indicate that lexical retrieval remains primary for SciFact and that hybrid reranking strategies can effectively leverage high-dimensional embedding space without requiring separate storage for diverse data types.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Retrieval & RAG

Related articles

arxiv arXiv cs.CL · 3d ago · 9 views

Nemotron-3-Embed closes commercial tax gap in MuSiQue retrieval benchmarks

A new audit of multi-hop retrieval benchmarks reveals that NVIDIA's Nemotron-3-Embed-8B has closed the performance gap between commercially licensed embedders and the non-commercial NV-Embed-v2 anchor on the MuSiQue benchmark. Released on July 16, 2026, this model is the only commercially licensed, free-to-self-host option that performs indistinguishably from the academic standard.

media Hugging Face Forums · 30d ago · 21 views

Author proposes open k registry for evidence saturation in local LLMs

The author has published a pilot paper and an open registry on Hugging Face to measure the evidence-saturation point (k*) of language models. This metric determines the optimal number of evidence fragments to inject into prompts, as adding more context does not monotonically improve correctness and can increase cost or epistemic contamination.

arxiv arXiv cs.CL · 36d ago · 8 views

LLM-as-a-judge signals fail to optimize closed-loop table recognition

A study using FinTabNet and OmniDocBench demonstrates that LLM-as-a-judge signals are insufficient for optimizing closed-loop regeneration in table recognition. The research reveals that judge scores frequently tied, rankings were not reproducible, and the system failed to recover better candidates produced by iteration.

arxiv arXiv cs.CL · 36d ago · 10 views

DeepStress stress-tests deep search agents against unreliable evidence

The authors introduce DeepStress, a framework for stress-testing deep search agents by replacing their retrieval modules with a controlled synthetic environment. This approach allows researchers to manipulate the frequency and quality of evidence across three dimensions: trustworthiness, relevance, and factuality.

arxiv arXiv cs.CL · 36d ago · 25 views

Hindcast closes data leakage in LLM forecaster evaluation

The Hindcast framework evaluates LLM forecasters by replaying resolved Polymarket prediction markets against a frozen snapshot of public Reddit, grading models as if they existed at a specific past date. This approach prevents two common channels of answer leakage: retrieval of post-event reports and training data contamination from events that occurred after the model's release.