A measurement study finds that 26 semantic post-hoc operators do not improve held-out accuracy over Best-of-N in frozen small code models. While two operators—expression-layer recovery and adaptive consensus early-stop—offer benefits in compute efficiency or program recovery, none outperform BoN in accuracy. The results highlight systemic limitations in error detection and coverage, suggesting that model harnesses and error coverage must be improved before post-hoc reasoning is considered.
arxiv
arXiv cs.CL
·
99d ago
·
research
Post-Hoc Operators Fail to Improve Accuracy in Small Code Models
from English
Importance 2/3
New harness with differentiators
arXiv cs.CL
DeepSeek
Code generation
Evaluation & benchmarks
Research paper
Related articles
arxiv
arXiv cs.AI
·
2d ago
·
18 views
CodeMidas scales agentic coding RL environments using source code
Researchers introduce CodeMidas, an agentic pipeline that constructs reinforcement learning environments from implemented functionality in open-source codebases, using source code as the sole input. The method allocates agentic compute to explore functionality, construct execution-grounded tests, and validate tasks through repeated rollouts, resulting in a dataset of 5,545 training tasks across 23 programming languages. Training MiMo-V2.5 on these tasks with GRPO improves performance on five diverse benchmarks, including DeepSWE (+11.7%), ProgramBench (+17%), and Terminal-Bench v2.1 (+8.5%). Trajectory analysis indicates the RL-trained agent exhibits better behaviors such as increased codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.
media
r/LocalLLaMA
·
82d ago
·
32 views
Follow-up: DeepSeek V4 Flash on 2x RTX PRO 6000 finishes real coding tasks faster than Sonnet and Opus, at about Sonnet quality
A follow-up benchmark evaluates DeepSeek V4 Flash running on two RTX PRO 6000 GPUs using vLLM, comparing its performance in real-world coding tasks against API-based models like Claude Sonnet and Opus. The study finds that while Opus and Fable maintain superior code quality, DeepSeek V4 Flash achieves approximately Sonnet-level quality with significantly faster wall-clock times.
arxiv
arXiv cs.CL
·
86d ago
·
3 views
The Signal-Coverage Matrix: Stratifying Type and Semantic Errors in Statement Autoformalization
This article introduces a signal-coverage matrix to stratify type and semantic errors in LLM autoformalization, moving beyond scalar type-correctness metrics. The framework categorizes outputs into true success, type-only, semantic-only, or both fail cells by crossing Lean elaborator results with semantic equivalence judgments.
arxiv
arXiv cs.AI
·
90d ago
·
1 view
Automated Semantic Fault Localization in SysML v2 Using Knowledge-Graph Augmented LLMs
This paper presents a human-in-the-loop framework for automatically identifying and repairing semantic errors in SysML v2 models that compilers cannot detect. The approach combines fine-tuned Small Language Models with a domain knowledge graph to ground repair suggestions in valid engineering constraints.
arxiv
arXiv cs.AI
·
91d ago
·
3 views
Context-Aware Distillation and Ablation for Text2DSL
A new Text2DSL system uses context-aware distillation with a structured context of BNF grammar, API specification, and closed identifier vocabulary. Ablation studies show that the vocabulary has the largest impact on semantic quality, while API and BNF significantly improve structural validity, confirming structured context as a critical, load-bearing component.