Zhipu's GLM-5.2 emerged as the top open-weight model, praised for its frontier-adjacent performance in daily use, with improvements in coding tasks and reduced 1M-token inference cost via IndexShare. It outperformed other open models in agentic knowledge work benchmarks, reaching 1266 Elo in Artificial Analysis' AA-Briefcase test, though only 3% of tasks were fully satisfied by top models, indicating persistent challenges in real-world long-horizon agent performance.
media
AI News (smol.ai)
·
49d ago
·
src: 52d ago
·
releases
GLM-5.2 Breakout and Open-Model Progress Highlighted
from English
Importance 2/3
New harness with differentiators
AI News (smol.ai)
Zhipu AI
Cognition
Cohere
AI agents
Code generation
Evaluation & benchmarks
Related articles
arxiv
arXiv cs.AI
·
52d ago
·
1 view
CAPRA: Multi-Agent LLM System for Software Architecture Feedback
CAPRA is a multi-agent LLM system that generates personalized, template-compliant LaTeX feedback on software architecture deliverables. It uses specialized agents, PyMuPDF, and gpt-4o to extract and analyze text and UML diagrams, with evidence anchoring and consistency management to ensure reliability. A preliminary evaluation of 10 student reports shows CAPRA met 88.8% of eight criteria and achieved moderate inter-rater agreement (kappa = 0.582), with each report processed in under 4 minutes.
arxiv
arXiv cs.LG
·
53d ago
ReproRepo: Scalable Reproducibility Audits with GitHub Issues
ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one human-reported blocker in 90% of 1,149 ML papers, highlighting their ability to detect visible failures and semantic issues, though exact localization remains limited.
arxiv
arXiv cs.CL
·
53d ago
ReproRepo: Scaling Reproducibility Audits with GitHub Issues
ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one semantically related blocker in 90% of paper-repository pairs without executing code.
arxiv
arXiv cs.AI
·
53d ago
ReproRepo: Scaling Reproducibility Audits with GitHub Issues
ReproRepo introduces a scalable framework using GitHub issues to evaluate ML paper reproducibility. It shows that LLM agents like Codex with GPT-5.5 identify at least one blocker in 90% of paper-repository pairs without executing code, though exact localization remains challenging.
arxiv
arXiv cs.CL
·
53d ago
·
6 views
GameCraft-Bench: Evaluating End-to-End Game Generation
GameCraft-Bench introduces a benchmark with 140 Godot tasks across 15 game families to assess coding agents' ability to generate playable games. Evaluations show the best agent achieves only 41.46% success, indicating significant challenges in producing complete, interactive games with coherent gameplay and visual feedback.