media r/LocalLLaMA · 1h ago · open_models

DeepSeek V4 Flash 0731 matches reported 82.7% on Terminal-Bench 2.1 in public harness

from English

An independent evaluation using the Ante 0.preview.71 harness confirms that DeepSeek V4 Flash 0731 achieves an 82.7% accuracy score on Terminal-Bench 2.1, matching DeepSeek's previously reported results.

The run consisted of 445 trials across 89 tasks with 5 trials per task and maximum reasoning effort.
The model achieved 368 successful trials out of 445, yielding an accuracy of 82.7% (±1.79 SE).
The evaluation used the `deepseek/deepseek-v4-flash-0731` model via OpenRouter with no skills enabled.
All configuration and trial records are publicly available in a Harbor job for verification.

The results demonstrate that DeepSeek V4 Flash 0731's performance is sensitive to the evaluation harness, providing useful data for benchmarking.

Importance 2/3 New harness with differentiators r/LocalLLaMA DeepSeek Evaluation & benchmarks

Benchmarks

Benchmark	Model	Score
Terminal-Bench 2	DeepSeek V4 Flash 0731	82.7%

Read original

arxiv arXiv cs.CL · 34d ago

Calibration reduces evaluator preference coupling in LLM agent feedback loops

A study investigates whether probability calibration of evaluator judgments can mitigate "evaluator preference coupling," where systematic biases propagate into an LLM agent's strategy. The authors present the first analysis of this mitigation approach, applying calibration to pairwise judgments to reduce spurious preference propagation.

arxiv arXiv cs.CL · 36d ago

SPLIT benchmark evaluates cross-lingual empathy and cultural grounding in English and Ukrainian LLM responses

The SPLIT benchmark is introduced to evaluate large language models' consistency in generating emotionally grounded responses across five crisis-related categories: Stress, Panic, Loneliness, Internal Displacement, and Tension. The framework assesses three technically diverse LLMs on empathetic accuracy, linguistic naturalness, and contextual & cultural grounding in both English and Ukrainian.

media r/LocalLLaMA · 37d ago · 2 views

Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality

A follow-up benchmark evaluates DeepSeek V4 Flash running on two RTX PRO 6000 GPUs using vLLM, comparing its performance in real-world coding tasks against API-based models like Claude Sonnet and Opus. The study finds that while Opus and Fable maintain superior code quality, DeepSeek V4 Flash achieves approximately Sonnet-level quality with significantly faster wall-clock times.

arxiv arXiv cs.CL · 40d ago

Not-quite-human tastes: the stylized omnivorousness of LLM survey surrogates

This study evaluates the ability of large-language models to approximate human cultural tastes by generating silicon surrogates from the Survey of Public Participation in the Arts. Using models from OpenAI, Anthropic, and DeepSeek, the authors analyze 277,470 synthetic respondents to determine if LLMs can faithfully replicate real-world survey data.

arxiv arXiv cs.CL · 40d ago · 2 views

OCR-VLMs Read Devanagari? Benchmark and Post-Correction Study

A study benchmarks ten OCR systems on Devanagari text, revealing that specialized OCR vision-language models are fragile under degradation and that strong English performance does not predict Indic script accuracy.