arxiv arXiv cs.AI · just now · src: 3d ago · research

GPT-5 and GPT-5 Nano match experts in microbial oncogenesis research appraisal

from English

A study demonstrates that GPT-5 and GPT-5 Nano achieve expert-level performance in extracting evidence and critically appraising research publications on microbial oncogenesis. Researchers benchmarked these models alongside Gemini 2.5 Pro and Gemini 2.5 Flash against domain experts using a dataset of 24 papers focused on MMTV-LV and breast cancer.

The evaluation used a structured template with 77 items across multiple question types, including MCQs, Likert scales, and free-text responses.
GPT-5 and GPT-5 Nano produced score distributions indistinguishable from human experts, while Gemini models were significantly more lenient in applying criteria.
Hallucinations were rare across all models, but methodological appraisal and contradiction identification within full texts remained persistent vulnerabilities.

The findings support the use of LLMs for automated systematic evidence synthesis, although further strengthening is needed for complex methodological tasks.

Importance 2/3 arXiv cs.AI Google DeepMind OpenAI Evaluation & benchmarks Research paper

Read original

arxiv arXiv cs.CL · 2h ago

GPT-5 and GPT-5 Nano match experts in microbial oncogenesis evidence extraction

A study benchmarking large language models on systematic evidence synthesis for microbial oncogenesis found that GPT-5 and GPT-5 Nano perform indistinguishably from domain experts. Researchers evaluated Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, and GPT-5 Nano on 24 research papers using a structured template of 77 items across multiple question types.

arxiv arXiv cs.CL · 38d ago

Automated grading of Linux/bash examinations using large language models

This study evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The research demonstrates that structured prompts significantly improve agreement with human graders, establishing a framework for AI-assisted assessment in computing education.

arxiv arXiv cs.CL · 48d ago · 2 views

AI Recommendation Ownership: Empirical Map of Brand Category Ownership

A study of 3,750 queries across five industries finds moderate recommendation concentration, with a mean Gini coefficient of 0.28. Cross-model agreement on top-recommended brands was only 41.6%, and displacement scores varied by industry, ranging from 0.4:1 to 4.3: 1. The results challenge the 'winner-takes-all' narrative and introduce three reproducible metrics for competitive-intelligence analysis.

arxiv arXiv cs.CL · 48d ago · 2 views

OpenBioRQ: Benchmark for Agentic Biomedical Research Faithfulness

OpenBioRQ introduces a benchmark of 12,553 unsolved biomedical research questions across 12 domains, designed to test agentic models' faithfulness and abstention. It evaluates models in a tool-using setting without answer keys, using real follow-up evidence rather than parametric knowledge, and reveals significant agentic collapse on the hardest questions where tools are no longer used despite being critical.

arxiv arXiv cs.CL · 48d ago · 2 views

Large Language Models Fail to Translate Fongbe Accurately

Evaluations show Fongbe translations achieve poor quality (1.0-2.2/5) compared to Hausa's acceptable scores (4.0-4.5/5), with a consistent 3x BLEU gap. Automatic metrics like BERTScore show embedding collapse and weak human correlation, especially for Hausa, while Gemini outperforms others for Fongbe and GPT-4o for Hausa in human judgments. Minimum sample sizes of 2,500 sentences are needed for stable model rankings.