media Hugging Face Forums · 8h ago · open_models

IntelligenceLab requests Long-Horizon-Terminal-Bench for Hub leaderboard

from English

IntelligenceLab has submitted a request to Hugging Face to add the Long-Horizon-Terminal-Bench (LHTB) dataset to the Benchmark allow-list. The goal is to enable the aggregation of community evaluation results for this benchmark on the Hugging Face Hub.

LHTB consists of 46 tasks designed to test how well LLM agents sustain useful work in a containerized terminal over hundreds of steps.
It utilizes hidden, rebuild-from-artifact verifiers instead of self-reported progress to assess agent performance.
Tasks are executed via the Harbor / Terminal-Bench 2.0 harness, with an eval.yaml file already present and validated in the repository.

Importance 1/3 Trust 1/3 Hugging Face Forums AI agents Evaluation & benchmarks

Read original

arxiv arXiv cs.LG · 5d ago

RELAI-VCL compounds optimization gains on Terminal-Bench 2.0

A study evaluates whether agent-optimization gains compound over time by testing three methods—GEPA, Meta Harness, and RELAI-VCL—on the hard tasks of Terminal-Bench 2.0. While all methods improve upon a static baseline, only RELAI-VCL successfully transfers to unseen tasks and continues improving after subsequent optimization rounds.

arxiv arXiv cs.CL · 5d ago

Set-shifting Behavioral Test for Harnessed Agents

Researchers introduce a benchmark to evaluate how LLM agents adapt when the reliability of available tools changes silently during an ongoing session. The framework mounts tool-skill libraries with redundancies and uses a branched schedule to shift the reliable tool group at hidden boundaries.

arxiv arXiv cs.CL · 5d ago

DevicesWorld introduces large-scale benchmark for cross-device agent collaboration

Researchers introduce DevicesWorld, a large-scale executable benchmark designed to evaluate LLM-based agents operating across heterogeneous environments including mobile, desktop, and IoT devices.

arxiv arXiv cs.CL · 5d ago

RELAI-VCL compounds agent-optimizer gains via continual learning

A study evaluates whether optimizer-driven gains compound over time by testing three agent-harness optimization methods on Terminal-Bench 2.0. The research compares GEPA, Meta Harness, and RELAI's Verifiable Continual Learning (RELAI-VCL) under identical budgets to determine if improvements persist as new tasks appear.

arxiv arXiv cs.AI · 5d ago

RELAI-VCL compounds agent-optimizer gains via continual learning

A study on Terminal-Bench 2.0 evaluates whether agent-optimization gains compound over time by comparing GEPA, Meta Harness, and RELAI's Verifiable Continual Learning (RELAI-VCL). The research tests if optimized agents can improve on new tasks without eroding previous gains.