media Hugging Face Forums · 1h ago · src: 3d ago · open_models

John6666 releases first independent reference implementation for REO v1.5

from English

John "John6666" has developed the first independent reference implementation of the Reference Evaluation Object (REO v1.5) protocol, fulfilling a key goal of the EPE research program to enable reproducible benchmarks by external contributors.

The repository includes the complete REO-20 benchmark, validation scripts, the official scorer, reproducibility tests, and evaluation schemas. It is structured as a reproducible evaluation package rather than a training dataset, allowing researchers to execute the benchmark under standardized conditions using open-weight models. A ready-to-run Google Colab notebook is also available to automate downloading the benchmark, validating files, running the 20 benchmark items with a fixed Qwen3.5-4B checkpoint, and producing a complete reproducibility package.

This implementation addresses criticisms that the EPE protocol was difficult for external researchers to reproduce, lowering the barrier for independent evaluation. Researchers can now run the benchmark themselves, compare different models, inspect the scorer, verify the protocol, and propose improvements or alternative implementations.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Open weights

Related articles

arxiv arXiv cs.CL · 18d ago

AGC-Bench introduces unified benchmark and AGC-Judge to measure artificial general creativity

Researchers introduce AGC-Bench, a unified benchmark for artificial general creativity constructed from 3,101 screened papers and covering 78 datasets across domains like brainstorming and STEM. To address bias in automated evaluation, the team fine-tunes Qwen3-30B on bias-corrected ratings to create AGC-Judge, an open-weight model that robustly scores new creativity benchmarks.

media r/LocalLLaMA · 20d ago · 2 views

Fine-tuned Gemma-4-31B for Copywriting Scores +290 Elo on EqBench3

A user has released a narrow fine-tune of the Gemma-4-31B-it model specifically optimized for copywriting and creative writing tasks. The model was trained to eliminate generic marketing clichés and adopt a direct-response style characterized by concrete specifics and tight calls to action.

arxiv arXiv cs.CL · 23d ago · 2 views

How Far Do On-Prem Open LLMs Get on Text-to-SQL? A Cross-Family Size x Technique Frontier on BIRD

This study evaluates the performance of open-weight large language models running on-premises for text-to-SQL tasks using a reproducible benchmark on the BIRD development split. It compares three model families across two generations while ablating specific accuracy-enhancing techniques to determine their actual value.

media r/LocalLLaMA · 23d ago

HuiHui abliterated model outperforms vanilla 3.6-35B-a3b on math and code

A custom quantization recipe applied to the HuiHui abliterated model demonstrates superior performance compared to the vanilla 3.6-35B-a3b variant in mathematics and coding tasks. The results suggest that removing refusal mechanisms allows the model to achieve greater accuracy and wisdom in these domains.

media r/LocalLLaMA · 25d ago

User questions existence of closed vs open LLM rankings and value of 70B-350B models

A Reddit user asks whether a solid leaderboard exists that compares closed-source and open-weight large language models side by side. They note that most available benchmarks feel fragmented and fail to address the practical differences between running models locally versus using API-based services.