MacAgentBench introduces a comprehensive benchmark with 676 tasks across 25 applications, 60% of which involve both GUI and CLI interactions. It uses deterministic rule-based evaluation and fine-grained multi-checkpoint scoring, revealing that Claude Opus 4.6 on OpenClaw achieves 73.7% Pass@1, primarily due to its skill library rather than framework design.
arxiv
arXiv cs.CL
·
47d ago
·
src: 49d ago
·
research
MacAgentBench Launches macOS AI Agent Benchmark
from English
Importance 2/3
arXiv cs.CL
OpenAI
Anthropic
Mistral AI
AI agents
Evaluation & benchmarks
Research paper
Related articles
arxiv
arXiv cs.CL
·
47d ago
·
2 views
OpenBioRQ: Benchmark for Agentic Biomedical Research Faithfulness
OpenBioRQ introduces a benchmark of 12,553 unsolved biomedical research questions across 12 domains, designed to test agentic models' faithfulness and abstention. It evaluates models in a tool-using setting without answer keys, using real follow-up evidence rather than parametric knowledge, and reveals significant agentic collapse on the hardest questions where tools are no longer used despite being critical.
arxiv
arXiv cs.AI
·
51d ago
·
1 view
Task Manager Reduces Queue Latency by 14-75% at Enterprise Scale
A Task Manager introduces priority inference, related-event merging, and preemption to enable continuous operation in enterprise AI. It reduces high-priority queue latency by 14-77% and improves related-event correctness by over 20 percentage points at enterprise scale, addressing agent discovery noise as the primary bottleneck.
arxiv
arXiv cs.LG
·
52d ago
·
1 view
TxBench-PP: AI Agent Performance in Preclinical Pharmacology
TxBench-PP is a verifiable benchmark for small-molecule preclinical pharmacology, testing AI agents' ability to derive accurate conclusions from real-world assay data. Across 16 model-harness configurations, no system reliably made correct preclinical pharmacology decisions, with the best performance at 59.3% (Claude Opus 4.8 / Pi) and 55.3% (GPT-5.5 / Pi) of endpoint attempts.
arxiv
arXiv cs.AI
·
52d ago
·
1 view
TxBench-PP: AI Agent Benchmark in Preclinical Pharmacology
TxBench-PP is a verifiable benchmark for small-molecule preclinical pharmacology, testing AI agents' ability to derive accurate conclusions from real-world assay data. Across 16 model configurations, no system reliably passed all evaluations, with the best performing setup (Claude Opus 4.8 / Pi) achieving 59.3% success rate on 300 endpoint attempts.
arxiv
arXiv cs.CL
·
53d ago
Routing Accuracy Degradation and Recovery in Enterprise Agent Systems
As enterprise agent tool catalogs scale from 10 to 110 agents, routing accuracy drops 16--23 percentage points on under-specified requests. An oracle analysis identifies retrieval and confusion gaps, with embedding-based shortlisting recovering +10--11pp F1. A human-annotated study of 1,435 utterances confirms real-world recovery of +10--17pp despite lower absolute performance.