blog Hamel Husain · 15h ago · evaluation

Anthropic releases auto eval tooling for Claude Code

from English

Anthropic has released new evaluation tooling for Claude Code, introducing a `claude-api` plugin with `build_eval` and `hill-climb` commands designed to help developers build evaluations, check graders, and improve applications against them.

The tool automatically discovers issues in conversation traces, such as problems with human handoff, formatting, and voice agents.
It allows users to create evaluators for specific failure modes, though the reviewer noted the initial scope was too broad.
The workflow includes commands to check graders and iteratively improve application performance based on evaluation results.

The author of the review suggests holding off on using the tool until it better supports data exploration before committing to an evaluator, noting that the current interface for reviewing labels is cumbersome.

Importance 2/3 New feature vs. leaders Hamel Husain Anthropic Code generation Evaluation & benchmarks

Read original

media r/LocalLLaMA · 90d ago · 33 views

Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality

A follow-up benchmark evaluates DeepSeek V4 Flash running on two RTX PRO 6000 GPUs using vLLM, comparing its performance in real-world coding tasks against API-based models like Claude Sonnet and Opus. The study finds that while Opus and Fable maintain superior code quality, DeepSeek V4 Flash achieves approximately Sonnet-level quality with significantly faster wall-clock times.

arxiv arXiv cs.CL · 91d ago · 21 views

TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

The authors introduce TestEvo-Bench, a live benchmark designed to evaluate how well test automation agents handle the co-evolution of code and tests. It addresses limitations in existing benchmarks by providing executable tasks anchored to real commit histories with environment configurations.

arxiv arXiv cs.AI · 94d ago · 8 views

Detecting AI Coding Agents in Open Source: A Validated Multi-Method Census of 180 Million Repositories

A multi-layered detection framework analyzing 180 million Git repositories reveals that single-signal methods significantly underestimate the prevalence of generative AI coding agents, missing up to 97% of activity. The study identifies over 320,000 commits per month from agents like Claude Code, which dominates silent adoption through configuration files rather than bot accounts.

media r/LocalLLaMA · 95d ago · 10 views

Qwen3.6 27B local vs Opus 4.8, voxel engine in raw C with zero frameworks

A comparison experiment pitted Claude Code on Opus 4.8 against a locally running Qwen3.6 27B model to build a voxel world engine in plain C without any external frameworks or libraries.

arxiv arXiv cs.CL · 101d ago · 10 views

P4IR Framework Improves LLM-Based Code Compliance Accuracy

P4IR, a two-stage framework, uses supervised fine-tuning and Group Relative Policy Optimization to enhance large language model-based automated code compliance systems. It reduces tree edit and token-level Levenshtein distances by up to 23.8% and 38.6% respectively, outperforming leading LLMs like Claude Opus, GPT-5.2, and GLM-4.7 in zero-shot settings with few-shot prompting, and reduces false positives by a statistically significant margin.