This work shows that Lean can serve as a symbolic process oracle, providing fine-grained, verified feedback during reinforcement learning. By parsing proof attempts into tactic sequences and using Lean's elaboration to mark sound steps and first failures, the system generates dense, type-theoretic reward signals. Experiments demonstrate tactic-level supervision outperforms outcome-only methods on benchmarks like MiniF2F and ProofNet, highlighting Lean's role as both evaluator and training reward source.
arxiv
arXiv cs.AI
·
51d ago
·
research
Lean as Process-Verified Reward Oracle in RL for Theorem Proving
from English
Importance 3/3
New feature vs. leaders
New harness with differentiators
arXiv cs.AI
DeepSeek
Google DeepMind
OpenAI
Evaluation & benchmarks
Reasoning models
Training methods
Related articles
arxiv
arXiv cs.LG
·
51d ago
·
1 view
VIMPO: Critic-Free Policy Optimization for LLMs
VIMPO introduces a critic-free policy optimization method that derives a policy-implied value function from KL-regularized reinforcement learning. It enables verifiable reward incorporation without training a critic and outperforms GRPO on mathematical benchmarks, especially under noisy rewards.
arxiv
arXiv cs.CL
·
51d ago
·
4 views
Bayesian Curriculum Learning on LLM Latent Manifolds
Manifold Bandits introduces Bayesian Manifold Curriculum (BMC), a framework that models problem sampling as a structured bandit problem in LLMs' latent space. BMC organizes tasks into a hierarchical tree and uses Bayesian learning to guide sampling, revealing tradeoffs between learning signal, task diversity, and evaluation relevance. Prioritizing difficulty alone fails to achieve strong downstream performance, underscoring the need for structure and type-aware sampling.
arxiv
arXiv cs.LG
·
52d ago
·
2 views
Discriminator-Guided RL Corrects Flow Matching with Data-Aligned Rewards
Discriminator-Guided RL (DRL) uses a pretrained representation space to train a discriminator that separates real data from model-generated samples. Its logit is used as a reward in KL-regularized RL, aligning model outputs with visual and semantic realism without human preferences. DRL improves FID and semantic FD across models like SiT and JiT, and enhances the Pareto frontier between preference and fidelity.
arxiv
arXiv cs.LG
·
52d ago
TAPO: Self-Distillation with Micro-Reflective Trajectories
TAPO advances self-distillation by constructing explicit micro-reflective trajectories that retain erroneous reasoning and insert natural-language diagnoses. These trajectories, derived from correct and incorrect model rollouts, provide fine-grained error corrections anchored in the model's own reasoning, improving both first-pass reasoning and error correction compared to GRPO.
arxiv
arXiv cs.AI
·
52d ago
·
1 view
Self-Conditioned Credit Assignment for RL with Verifiable Rewards
SC-GRPO uses per-token KL divergence from self-conditioned trajectories to weight gradients in reinforcement learning. It outperforms GRPO by 8.1% and DAPO by 5.9% across math, code, and agentic tasks, with superior out-of-distribution performance and better results than OPD.