arxiv arXiv cs.CL · 1h ago · src: 3d ago · research

GPT-5 and GPT-5 Nano match experts in microbial oncogenesis evidence extraction

from English

A study benchmarking large language models on systematic evidence synthesis for microbial oncogenesis found that GPT-5 and GPT-5 Nano perform indistinguishably from domain experts. Researchers evaluated Gemini 2.5 Pro, Gemini 2.5 Flash, GPT-5, and GPT-5 Nano on 24 research papers using a structured template of 77 items across multiple question types.

LLM responses aligned closely with experts across all question types, including MCQ, Likert-scale, multi-select, and free-text formats.
GPT-5 and GPT-5 Nano achieved score distributions statistically indistinguishable from human experts.
Gemini models behaved similarly but were significantly more lenient in applying microbial oncogenesis criteria.
Hallucinations were rare, though methodological appraisal and contradiction identification remained persistent vulnerabilities for all models.

The results support the use of LLMs for automated systematic evidence synthesis to identify microbe-cancer pairs, although further strengthening is needed for complex full-text analysis tasks.

Importance 2/3 arXiv cs.CL Google DeepMind OpenAI Evaluation & benchmarks Research paper

Related articles

arxiv arXiv cs.CL · 38d ago

Automated grading of Linux/bash examinations using large language models

This study evaluates whether four frontier Large Language Models (GPT, Claude Opus, Gemini, and GLM) can approximate expert judgment when grading short Linux/bash command responses. The research demonstrates that structured prompts significantly improve agreement with human graders, establishing a framework for AI-assisted assessment in computing education.

arxiv arXiv cs.CL · 48d ago · 2 views

AI Recommendation Ownership: Empirical Map of Brand Category Ownership

A study of 3,750 queries across five industries finds moderate recommendation concentration, with a mean Gini coefficient of 0.28. Cross-model agreement on top-recommended brands was only 41.6%, and displacement scores varied by industry, ranging from 0.4:1 to 4.3: 1. The results challenge the 'winner-takes-all' narrative and introduce three reproducible metrics for competitive-intelligence analysis.

arxiv arXiv cs.CL · 48d ago · 2 views

OpenBioRQ: Benchmark for Agentic Biomedical Research Faithfulness

OpenBioRQ introduces a benchmark of 12,553 unsolved biomedical research questions across 12 domains, designed to test agentic models' faithfulness and abstention. It evaluates models in a tool-using setting without answer keys, using real follow-up evidence rather than parametric knowledge, and reveals significant agentic collapse on the hardest questions where tools are no longer used despite being critical.

arxiv arXiv cs.CL · 48d ago · 2 views

Large Language Models Fail to Translate Fongbe Accurately

Evaluations show Fongbe translations achieve poor quality (1.0-2.2/5) compared to Hausa's acceptable scores (4.0-4.5/5), with a consistent 3x BLEU gap. Automatic metrics like BERTScore show embedding collapse and weak human correlation, especially for Hausa, while Gemini outperforms others for Fongbe and GPT-4o for Hausa in human judgments. Minimum sample sizes of 2,500 sentences are needed for stable model rankings.

arxiv arXiv cs.LG · 52d ago

LLM Alignment Using Implicit User Feedback

A new dataset, IFLLM, collects mouse trajectories and eye gazing data from users interacting with LLMs. It shows that implicit feedback significantly improves LLM alignment, boosting text-based reward model accuracy from 55% to 64% and nearly tripling response quality improvements after DPO training on eight LLMs.