media Hugging Face Forums · 1h ago · open_models

AbstractPhil releases Beatrix V3, a 376M byte-level model with splat memory

from English

AbstractPhil has released Beatrix V3, a 376 million parameter byte-level language model that utilizes "splat memory" in every attention block. The model features a vocabulary of 256 and a context window of 4,096 bytes, trained on 64.4 billion bytes over two RTX 5090 cards.

The architecture achieves 0.9861 bits per byte on held-out web text without loss spikes.
It includes detachable 13.7 million parameter adapters after every block to handle specific curriculum stages.
The model demonstrates strong text encoding capabilities, matching T5 performance in its middle blocks.
Splat memory shows limitations in long-distance recall compared to softmax controls, prompting plans for a hybrid approach.

The release includes the model weights and technical documentation, inviting feedback on architectural improvements such as hybrid attention mechanisms and forgetful memory modules.

Importance 1/3 Trust 1/3 Hugging Face Forums Hugging Face Research paper Training methods

Read original

arxiv arXiv cs.AI · 22h ago · 1 view

On-policy power distillation sharpens language model reasoning without search

Researchers introduce On-Policy Power Distillation (OPPD), a method that trains models to generate high-quality answers in a single pass by mimicking the sharpened probability distribution of a teacher model. This approach eliminates the need for extensive candidate sampling during inference while significantly boosting performance on mathematical and coding benchmarks.

arxiv arXiv cs.LG · 1d ago · 1 view

On-policy power distillation sharpens reasoning without search

Researchers introduce On-Policy Power Distillation (OPPD), a method that trains language models to generate high-quality answers in a single pass by mimicking the sharpened probability distribution of a teacher model. This approach eliminates the need for extensive candidate sampling during inference while significantly boosting performance on reasoning tasks.

media r/LocalLLaMA · 4d ago · 29 views

Hugging Face publishes guide to multi-harness RL training

Lewis from the Hugging Face post-training team has published a comprehensive guide on training open models using different coding harnesses. The article details how the team solved this challenge by leveraging open source libraries such as TRL and the Harbor framework for reinforcement learning environments.

arxiv arXiv cs.LG · 15d ago · 40 views

Fathom-Vaidya improves medical reasoning with rubric-based rewards

The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance diagnostic and clinical healthcare reasoning. The training framework first applies rule-guided RL to MedBullets-derived questions for diagnosis, then utilizes 5.3k synthetic multi-turn scenarios with multi-dimensional rubrics for interactive clinical tasks.

arxiv arXiv cs.AI · 15d ago · 41 views

Fathom-Vaidya improves medical reasoning with rubric-based rewards

The authors introduce Fathom-Vaidya, a 30B parameter model that uses synthetic data and rubric-based reinforcement learning to enhance both diagnostic and clinical healthcare reasoning. The training framework first targets diagnostic accuracy using MedBullets-derived questions and then addresses multi-turn clinical interactions through 5.3k generated scenarios with multi-dimensional rubrics.