media Hugging Face Forums · 3d ago · open_models

Proposal: Progressive knowledge transfer from Qwen 9B to 27B for a fixed 4B model

from English

A user on the Hugging Face forums proposes an experiment in progressive knowledge transfer, where a fixed-size student model (Qwen 4B) learns sequentially from increasingly larger teacher models rather than directly from the largest available one.

The process involves the 4B model learning from Qwen 8B or 9B, then discarding that teacher to learn from Qwen 12B, and subsequently from Qwen 27B.
This approach mimics human learning by building on previously acquired knowledge instead of attempting to bridge a large capability gap in a single step.
The proposal suggests using a method similar to the Heretic project, employing a "search and reinforce" strategy to identify and add useful capabilities while avoiding rediscovery of already learned traits.
Success would be measured by whether the fixed-size model improves measurably and preserves earlier gains, offering insight into the practical knowledge capacity of small models.

The author suggests this method could determine if gradual accumulation of knowledge is more effective than direct distillation from the largest teacher for small models.

Importance 1/3 Trust 1/3 Hugging Face Forums Research paper Training methods

Related articles

arxiv arXiv cs.LG · 1d ago · 9 views

Fathom-Vaidya improves medical reasoning with rubric-based rewards

The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance diagnostic and clinical healthcare reasoning. The training framework first applies rule-guided RL to MedBullets-derived questions for diagnosis, then utilizes 5.3k synthetic multi-turn scenarios with multi-dimensional rubrics for interactive clinical tasks.

arxiv arXiv cs.AI · 1d ago · 12 views

Fathom-Vaidya improves medical reasoning with rubric-based rewards

The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance both diagnostic and clinical healthcare reasoning. The training framework first targets diagnostic accuracy using MedBullets-derived questions and then addresses multi-turn clinical interactions through 5.3k generated scenarios with multi-dimensional rubrics.

arxiv arXiv cs.CL · 1d ago · 12 views

Fathom-Vaidya improves medical reasoning with rubric-based rewards

The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance both diagnostic and clinical healthcare reasoning.

media Hugging Face Forums · 2d ago · 18 views

MB-Bidram shares experimental WideNDepth architecture separating reasoning from knowledge

MB-Bidram has released an experimental neural architecture called WideNDepth (WND) designed to separate reasoning capabilities from knowledge storage. The model consists of a "Wide part" that functions as memory and a "Depth part" that acts as the reasoner.

arxiv arXiv cs.LG · 7d ago · 20 views

OpenAI releases Open-1B with fully auditable training run

OpenAI has released Open-1B, a language model trained under a regime where every operation during training is independently reproducible on heterogeneous commodity hardware with bitwise certainty. This approach addresses the reproducibility issues inherent in open-source models by imposing a definite order on sources of nondeterminism, such as GPU kernel reductions and data batch ordering.