A new Text2DSL system uses context-aware distillation with a structured context of BNF grammar, API specification, and closed identifier vocabulary. Ablation studies show that the vocabulary has the largest impact on semantic quality, while API and BNF significantly improve structural validity, confirming structured context as a critical, load-bearing component.
arxiv
arXiv cs.AI
·
91d ago
·
src: 94d ago
·
research
Context-Aware Distillation and Ablation for Text2DSL
from English
Related articles
arxiv
arXiv cs.CL
·
86d ago
·
3 views
The Signal-Coverage Matrix: Stratifying Type and Semantic Errors in Statement Autoformalization
This article introduces a signal-coverage matrix to stratify type and semantic errors in LLM autoformalization, moving beyond scalar type-correctness metrics. The framework categorizes outputs into true success, type-only, semantic-only, or both fail cells by crossing Lean elaborator results with semantic equivalence judgments.
arxiv
arXiv cs.AI
·
90d ago
·
1 view
Automated Semantic Fault Localization in SysML v2 Using Knowledge-Graph Augmented LLMs
This paper presents a human-in-the-loop framework for automatically identifying and repairing semantic errors in SysML v2 models that compilers cannot detect. The approach combines fine-tuned Small Language Models with a domain knowledge graph to ground repair suggestions in valid engineering constraints.
arxiv
arXiv cs.CL
·
99d ago
·
3 views
Post-Hoc Operators Fail to Improve Accuracy in Small Code Models
A measurement study finds that 26 semantic post-hoc operators do not improve held-out accuracy over Best-of-N in frozen small code models. While two operators—expression-layer recovery and adaptive consensus early-stop—offer benefits in compute efficiency or program recovery, none outperform BoN in accuracy. The results highlight systemic limitations in error detection and coverage, suggesting that model harnesses and error coverage must be improved before post-hoc reasoning is considered.
arxiv
arXiv cs.AI
·
2d ago
·
18 views
CodeMidas scales agentic coding RL environments using source code
Researchers introduce CodeMidas, an agentic pipeline that constructs reinforcement learning environments from implemented functionality in open-source codebases, using source code as the sole input. The method allocates agentic compute to explore functionality, construct execution-grounded tests, and validate tasks through repeated rollouts, resulting in a dataset of 5,545 training tasks across 23 programming languages. Training MiMo-V2.5 on these tasks with GRPO improves performance on five diverse benchmarks, including DeepSWE (+11.7%), ProgramBench (+17%), and Terminal-Bench v2.1 (+8.5%). Trajectory analysis indicates the RL-trained agent exhibits better behaviors such as increased codebase exploration and more diverse self-verification. These results establish source code as a scalable foundation for constructing RL environments that improve coding agents across diverse software tasks.
media
Hugging Face Forums
·
3d ago
·
16 views
Researcher seeks one independent annotator to resolve LLM agreement ambiguity
A researcher is requesting a single independent human annotator to label 100 Turkish narrative scenes in order to determine whether low inter-rater agreement stems from the interpretive nature of the task or from underspecified annotation definitions. The study found that four LLMs and a rule-based detector agreed with each other and the human reference at roughly chance level, with Cohen’s κ scores ranging from 0.000 to 0.185.