lab OpenAI · 726d ago

OpenAI releases SWE-bench Verified with human-validated subset

from English

OpenAI has released SWE-bench Verified, a human-validated subset of the SWE-bench benchmark designed to more reliably evaluate AI models' ability to solve real-world software issues. This new version supersedes the original SWE-bench and SWE-bench Lite test sets by addressing flaws that previously led to underestimating model capabilities.

The release includes 500 samples verified by professional developers to ensure well-specified issue descriptions and appropriately scoped unit tests. It also provides annotations for all 1,699 sampled test items, enabling difficulty slicing into 'easy' (196 tasks) and 'hard' (45 tasks) subsets. Additionally, a new evaluation harness using containerized Docker environments is released to improve setup reliability.

On SWE-bench Verified, GPT-4o resolves 33.2% of samples, while the best open-source scaffold, Agentless, doubles its previous score to 16%.

Importance 1/3 New harness with differentiators OpenAI OpenAI Benchmark results Evaluation & benchmarks

Benchmarks

Benchmark	Model	Score
SWE-bench Verified	GPT‑4o	33.2%
SWE-bench Verified	Agentless	32%

Read original

arxiv arXiv cs.CL · 35d ago

Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy

A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.

arxiv arXiv cs.CL · 51d ago · 2 views

Benchmarking Agentic Review Systems for AI-Assisted Research

A study evaluates four AI review systems across six language models, finding OpenAIReview with GPT-5.5 achieves 83.0% accuracy in matching paper quality to external signals and detects 71.6% of injected errors. Real user feedback shows positive sentiment, with a 1.44-to-1 vote ratio, though false positives and minor nitpicks remain common.

arxiv arXiv cs.CL · 51d ago · 1 view

REDACT: Multilingual PII Benchmark with Systematic Control

REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.

media r/LocalLLaMA · 51d ago · 1 view

GLM-5.2 Outperforms GPT-5.5 in AA-Briefcase Evaluation

Artificial Analysis' new agentic knowledge work evaluation, AA-Briefcase, shows GLM-5.2 surpassing GPT-5.5 in performance. The benchmark assesses real-world task execution and reasoning capabilities in knowledge work scenarios.

arxiv arXiv cs.AI · 52d ago

Multi-Domain Benchmark for Detecting AI-Generated Text-Rich Images

A new benchmark evaluates AI-generated text-rich images across six domains, including commercial posters and receipts. It reveals significant domain-dependent performance and sensitivity to JPEG compression, highlighting the need for text- and layout-aware detection methods.