TxBench-PP is a verifiable benchmark for small-molecule preclinical pharmacology, testing AI agents' ability to derive accurate conclusions from real-world assay data. Across 16 model-harness configurations, no system reliably made correct preclinical pharmacology decisions, with the best performance at 59.3% (Claude Opus 4.8 / Pi) and 55.3% (GPT-5.5 / Pi) of endpoint attempts.
arxiv
arXiv cs.LG
·
52d ago
·
research
TxBench-PP: AI Agent Performance in Preclinical Pharmacology
from English
Importance 3/3
arXiv cs.LG
Anthropic
Mistral AI
OpenAI
AI agents
Evaluation & benchmarks
Reasoning models
Related articles
arxiv
arXiv cs.AI
·
52d ago
·
1 view
TxBench-PP: AI Agent Benchmark in Preclinical Pharmacology
TxBench-PP is a verifiable benchmark for small-molecule preclinical pharmacology, testing AI agents' ability to derive accurate conclusions from real-world assay data. Across 16 model configurations, no system reliably passed all evaluations, with the best performing setup (Claude Opus 4.8 / Pi) achieving 59.3% success rate on 300 endpoint attempts.
arxiv
arXiv cs.CL
·
47d ago
MacAgentBench Launches macOS AI Agent Benchmark
MacAgentBench introduces a comprehensive benchmark with 676 tasks across 25 applications, 60% of which involve both GUI and CLI interactions. It uses deterministic rule-based evaluation and fine-grained multi-checkpoint scoring, revealing that Claude Opus 4.6 on OpenClaw achieves 73.7% Pass@1, primarily due to its skill library rather than framework design.
arxiv
arXiv cs.AI
·
51d ago
·
3 views
Trajectory Mining Reveals Skill Structure but Fails to Improve Policies
A three-stage pipeline mines skill libraries from GUI interaction data, achieving high purity in five of eight clusters against InteraSkill labels. However, the method only slightly improves skill-step accuracy on IW and fails to advance performance on BrowseComp+ or key metrics, indicating limitations in cross-domain policy transfer.
arxiv
arXiv cs.CL
·
51d ago
·
2 views
AtomMem: Simple and Effective Memory System for LLM Agents
AtomMem introduces a memory system that stores high-value atomic facts from long-form interactions. It uses hierarchical event structures and temporal profiles to capture coherent episodic contexts and track evolving user attributes, enabling stable and efficient memory evolution. Experiments on the LoCoMo benchmark show AtomMem achieves state-of-the-art performance in reasoning tasks.
arxiv
arXiv cs.CL
·
51d ago
GEMS: Geometric Constraints Enable Multi-Semantic Superposition in LLMs
GEMS enables training-free superposition of multiple semantic directions in LLMs by addressing distributional deviation and directional interference through geometric constraints. On GSM8K, it maintains 98% accuracy with three non-mathematical directions, while unconstrained addition drops to 4%; on Wikitext-2, it increases PPL by only 2.2%.