The authors introduce iLLaDA, an 8B parameter masked diffusion language model trained from scratch using fully bidirectional attention. This approach contrasts with the predominant autoregressive factorization and causal attention used in modern large language models. The model's pre-training scaled to 12 trillion tokens, followed by supervised fine-tuning on a 25 billion-token instruction corpus for 12 epochs. iLLaDA maintains the masked diffusion objective throughout both training phases and employs variable-length generation for efficiency. It also introduces confidence-based scoring to enhance performance on multiple-choice evaluation tasks. Benchmark results show significant improvements over its predecessor, LLaDA, including gains of 21.6 points on BBH and 14.9 points on ARC-Challenge for the base model. The instruction-tuned variant achieved increases of 14.5 points on MATH and 16.5 points on HumanEval. Despite its non-autoregressive nature, iLLaDA remains competitive with Qwen2.5 7B across several metrics.
iLLaDA: An 8B Masked Diffusion Language Model with Fully Bidirectional Attention
Benchmarks
| Benchmark | Model | Score |
|---|---|---|
| ARC-AGI 1 | iLLaDA-Base | 14.9pts |
| HumanEval+ | iLLaDA-Instruct | 16.5pts |
Fathom-Vaidya improves medical reasoning with rubric-based rewards
The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance diagnostic and clinical healthcare reasoning. The training framework first applies rule-guided RL to MedBullets-derived questions for diagnosis, then utilizes 5.3k synthetic multi-turn scenarios with multi-dimensional rubrics for interactive clinical tasks.
Fathom-Vaidya improves medical reasoning with rubric-based rewards
The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance both diagnostic and clinical healthcare reasoning. The training framework first targets diagnostic accuracy using MedBullets-derived questions and then addresses multi-turn clinical interactions through 5.3k generated scenarios with multi-dimensional rubrics.
Fathom-Vaidya improves medical reasoning with rubric-based rewards
The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance both diagnostic and clinical healthcare reasoning.
MB-Bidram shares experimental WideNDepth architecture separating reasoning from knowledge
MB-Bidram has released an experimental neural architecture called WideNDepth (WND) designed to separate reasoning capabilities from knowledge storage. The model consists of a "Wide part" that functions as memory and a "Depth part" that acts as the reasoner.
Proposal: Progressive knowledge transfer from Qwen 9B to 27B for a fixed 4B model
A user on the Hugging Face forums proposes an experiment in progressive knowledge transfer, where a fixed-size student model (Qwen 4B) learns sequentially from increasingly larger teacher models rather than directly from the largest available one.