JAMER introduces JamSet and JamBench, the first project-level game code dataset and benchmark on a professional game engine. Built from 8,133 verified Game Jam projects, it enables deterministic evaluation and reveals a capability cliff in AI models as project scale increases, with runtime pass rates dropping from 80.4% to 5.7%.
arxiv
arXiv cs.CL
·
51d ago
·
research
JAMER: Project-Level Code Framework Dataset and Benchmark
from English
Related articles
arxiv
arXiv cs.CL
·
51d ago
·
1 view
Tool-Intent Stabilization in Streaming RAG
A study measures tool-intent stabilization in Streaming RAG, defining when speculative tool queries converge to correct answers. On the CRAG benchmark, 73.9% of queries allow substantial latency hiding, with early stabilization observed in questions with verbatim retrievable evidence. Question type significantly predicts early versus late stabilization, informing when speculative triggers are effective.
arxiv
arXiv cs.AI
·
52d ago
·
1 view
Data Intelligence Agents Enable Autonomous Data Querying
Data Intelligence Agents (DIA) deploy autonomous coding agents to streamline enterprise data workflows. The Query Generator matches or exceeds top published results on seven SQL benchmarks across four dialects, showing generalization through natural-language instructions and execution-based architecture.
arxiv
arXiv cs.AI
·
52d ago
·
1 view
CAPRA: Multi-Agent LLM System for Software Architecture Feedback
CAPRA is a multi-agent LLM system that generates personalized, template-compliant LaTeX feedback on software architecture deliverables. It uses specialized agents, PyMuPDF, and gpt-4o to extract and analyze text and UML diagrams, with evidence anchoring and consistency management to ensure reliability. A preliminary evaluation of 10 student reports shows CAPRA met 88.8% of eight criteria and achieved moderate inter-rater agreement (kappa = 0.582), with each report processed in under 4 minutes.
arxiv
arXiv cs.CL
·
53d ago
·
2 views
SwiftTrans Improves LLM Code Translation Efficiency
SwiftTrans addresses runtime efficiency gaps in LLM-based code translation by introducing Multi-Perspective Exploration and Difference-Aware Selection. The framework extends CodeNet, F2SBench, and introduces SwiftBench to evaluate runtime performance, showing consistent improvements in both correctness and efficiency across benchmarks.
arxiv
arXiv cs.CL
·
47d ago
·
1 view
CAT-Translate: Compact Japanese-English Models Outperform Multilingual Ones in Real-World Tasks
CAT-Translate introduces a family of small, open-source models specialized for Japanese-English translation. Using synthetic parallel corpora and a two-stage fine-tuning approach, the models achieve superior performance on real-world benchmarks across business, legal, medical, financial, and patent domains, outperforming large multilingual models in practical applications.