arxiv arXiv cs.AI · 2h ago · research

OpenForgeRL enables end-to-end training of harness-based agents in any environment

from English

OpenForgeRL is an open-source framework that allows researchers to train harness-based AI agents end-to-end using standard reinforcement learning stacks. It achieves this by decoupling training from inference through a lightweight proxy that records model calls as data and a Kubernetes orchestrator that manages remote container rollouts.

The system supports diverse environments, including tool/claw-based agents and multimodal GUI browser agents.
OpenForgeClaw achieved 31.7 pass^3 on ClawEval and 33.7 on QwenClawBench using only hundreds to a few thousand tasks.
OpenForgeGUI reached 37.7 on OSWorld-Verified, 63.0 on Online-Mind2Web, and 72.3 on WebVoyager.
Both variants outperform open baselines of similar size and match or surpass larger models in GUI settings.
Analysis reveals that harness choice significantly impacts learning difficulty and that RL improves reliability metrics like self-verification.

The framework enables agents to be trained directly in the real harnesses and environments where they are deployed, facilitating easier study and improvement of agentic behavior.

Importance 2/3 New harness with differentiators arXiv cs.AI OpenAI AI agents Training methods

Benchmarks

Benchmark	Model	Score
OSWorld	OpenForgeGUI	37.7%
SWE-bench Pro	OpenForgeClaw	31.7%

Related articles

arxiv arXiv cs.LG · 35d ago

Marginal Advantage Accumulation for Memory-Driven Agent Self-Evolution

This paper introduces Marginal Advantage Accumulation (MAA), a post-processing architecture that addresses cross-batch inconsistency in memory-driven agent self-evolution. MAA formalizes alignment and comparability as structural conditions, uses differential signals and exponential moving average to accumulate signed evidence per operation, and ensures traceability via semantic identity merging. It outperforms batch-level baselines in 14 out of 16 settings and reduces token consumption by about 75%.

arxiv arXiv cs.LG · 35d ago

LLM-based Hierarchical Control in Multi-Agent Games

A hierarchical system using a pretrained LLM to select RL skill policies outperforms flat RL in a 2v2 King of the Hill environment. It matches hand-crafted behavior tree performance in win rate and is perceived as more human-like by 60% of users, highlighting effective coordination and adaptability without manual rule design.

arxiv arXiv cs.CL · 36d ago

GraphPO: Graph-based Policy Optimization for Reasoning Models

GraphPO introduces a directed acyclic graph framework to represent reasoning rollouts, merging semantically equivalent paths to reduce redundant exploration. It assigns efficiency and correctness advantages to edges, improving inference efficiency and process supervision while reducing advantage-estimation variance. Experiments show GraphPO outperforms chain- and tree-based methods on three LLMs across reasoning and agentic search tasks under identical token or response budgets.

arxiv arXiv cs.AI · 36d ago

RODS: Reward-Driven Online Data Synthesis for Multi-Turn Tool-Use Agents

RODS addresses sample depletion in multi-turn tool-use RL by using reward variance to detect capability boundaries. It synthesizes new data in real time, matching structural complexity of boundary samples, and maintains a dynamic replay buffer that co-evolves with the policy. RODS achieves performance comparable to a 17K-sample offline pipeline with 20x fewer trajectories.

arxiv arXiv cs.LG · 37d ago

Compositional Generalization in Language Model Reasoning

A hierarchical latent selection model shows that supervised fine-tuning and reinforcement learning work together to enable compositional generalization in language models. SFT provides raw module materials, while RL identifies and recombines atomic modules from compound traces to solve new problems. Training on compound traces leads to stronger generalization than isolated module training, and an effective protocol is found where SFT ensures module coverage and RL drives exploration of novel compositions.