ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one human-reported blocker in 90% of 1,149 ML papers, highlighting their ability to detect visible failures and semantic issues, though exact localization remains limited.
arxiv
arXiv cs.LG
·
53d ago
·
research
ReproRepo: Scalable Reproducibility Audits with GitHub Issues
from English
Importance 2/3
arXiv cs.LG
OpenAI
Cohere
Mistral AI
AI agents
Code generation
Evaluation & benchmarks
Related articles
arxiv
arXiv cs.CL
·
53d ago
ReproRepo: Scaling Reproducibility Audits with GitHub Issues
ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one semantically related blocker in 90% of paper-repository pairs without executing code.
arxiv
arXiv cs.AI
·
53d ago
ReproRepo: Scaling Reproducibility Audits with GitHub Issues
ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one blocker in 90% of paper-repository pairs without executing code, though exact localization remains challenging.
arxiv
arXiv cs.CL
·
53d ago
·
6 views
GameCraft-Bench: Evaluating End-to-End Game Generation
GameCraft-Bench introduces a benchmark with 140 Godot tasks across 15 game families to assess coding agents' ability to generate playable games. Evaluations show the best agent achieves only 41.46% success, indicating significant challenges in producing complete, interactive games with coherent gameplay and visual feedback.
arxiv
arXiv cs.AI
·
51d ago
·
3 views
Trajectory Mining Reveals Skill Structure but Fails to Improve Policies
A three-stage pipeline mines skill libraries from GUI interaction data, achieving high purity in five of eight clusters against InteraSkill labels. However, the method only slightly improves skill-step accuracy on IW and fails to advance performance on BrowseComp+ or key metrics, indicating limitations in cross-domain policy transfer.
arxiv
arXiv cs.AI
·
52d ago
·
1 view
CAPRA: Multi-Agent LLM System for Software Architecture Feedback
CAPRA is a multi-agent LLM system that generates personalized, template-compliant LaTeX feedback on software architecture deliverables. It uses specialized agents, PyMuPDF, and gpt-4o to extract and analyze text and UML diagrams, with evidence anchoring and consistency management to ensure reliability. A preliminary evaluation of 10 student reports shows CAPRA met 88.8% of eight criteria and achieved moderate inter-rater agreement (kappa = 0.582), with each report processed in under 4 minutes.