lab NVIDIA Research · 15h ago · research

ZPPO uses prompts instead of gradients to improve small vision-language model training

from English

Researchers introduce Zone of Proximal Policy Optimization (ZPPO), a method that injects teacher knowledge into prompts rather than policy gradients to address brittleness in knowledge distillation and drift in reinforcement learning. ZPPO constructs Binary Candidate-included Questions (BCQ) and Negative Candidate-included Questions (NCQ) for hard questions, recirculating them via a replay buffer until the student's accuracy improves or they are evicted.

ZPPO pairs one correct teacher response with an incorrect student response in BCQs to force discrimination, while NCQs aggregate student failures to surface shared error modes.
The method was tested on the Qwen3.5 family at scales from 0.8B to 9B using a 27B teacher model.
Evaluations covered a 31-benchmark suite including 16 VLM, 10 LLM, and 5 Video tasks.
ZPPO outperformed off/on-policy distillation and GRPO, with the largest gains observed at the smallest model scale.

By keeping the teacher inside the prompt rather than the policy gradient, ZPPO avoids breaking on-policy assumptions while amplifying learning within the student's current zone of proximal development.

Importance 1/3 Trust 3/3 NVIDIA Research Research paper Training methods

Related articles

lab NVIDIA Research · 14h ago · 2 views

Masters distills vision-language models via masking teacher and reinforcing student

Researchers propose Masters, a mask-progressive reinforcement learning framework for distilling large-scale vision-language models into compact versions suitable for edge deployment. The method addresses the instability caused by the size gap between teacher and student models.

arxiv arXiv cs.LG · 5d ago

U-OPSD enables unsupervised on-policy self-distillation for LLMs

Researchers propose Unsupervised On-Policy Self-Distillation (U-OPSD), a method that achieves post-training improvement for large language models using only the model's own generations without external supervision. The approach samples multiple rollouts to construct a pseudo-solution via majority vote, then distills a teacher distribution into prefixes of the model's longest incorrect completion.

arxiv arXiv cs.CL · 12d ago · 3 views

Memory Decoder at Scale: Scaling parametric long-term memory to 6.9B parameters

Researchers present Memory Decoder at Scale, a system that scales parametric long-term memory models up to 6.9B parameters and pretrains them on 300B tokens. To address the infeasibility of standard Faiss pipelines at this data scale, the authors implement a distributed indexing pipeline with sparse, batch-wise loading of kNN distributions.

arxiv arXiv cs.CL · 14d ago

Relay-OPD reduces training trajectory length by over 50% in on-policy distillation

Researchers introduce Relay On-Policy Distillation (Relay-OPD) to address prefix failure in standard on-policy distillation, where student errors lead to unreliable supervision. The method uses a teacher-student continuation asymmetry as a trigger for the teacher to briefly take over and redirect the trajectory before the student resumes.

media Hugging Face Forums · 18d ago

ThetaScan v0.1 releases open scan-parallel nonlinear memory with 17M LM results

A research preview of ThetaScan v0.1, a fixed-state nonlinear token mixer, has been released as an open-source implementation. The architecture ensures that every token computes its memory write from the current input and shared parameters rather than an evolving fast state, allowing writes to compose through an associative prefix scan.