media Hugging Face Forums · 1d ago · open_models

Jen Wei debugs Muon and Dion3 optimizer issues in OLMo-core

from English

Jen Wei encountered a learning rate death spiral while testing AdamW, Muon, and the latest Dion3 implementation within OLMo-core for an upcoming PyTorch Conference talk.

The debugging process revealed issues involving distributed optimizer state and LR scheduling.
A specific PyTorch tensor aliasing trap was identified as a key cause of the instability.
Wei documented the full debugging adventure on Medium to help others avoid similar memory traps in distributed optimizers.

Importance 1/3 Trust 1/3 Hugging Face Forums Inference efficiency Training methods

Read original

lab Hugging Face Blog · 9d ago · 16 views

TRL v1.14 AsyncGRPOTrainer enables LoRA-only sync across Hugging Face Jobs

TRL v1.14 introduces LoRA support to the AsyncGRPOTrainer, allowing it to train a Low-Rank Adaptation adapter and synchronize only that small file to vLLM inference workers. This architecture decouples training and generation jobs on separate machines by using a shared Storage Bucket as a filesystem bridge, eliminating the need for NCCL or direct network communication between nodes.

media Hugging Face Forums · 16d ago · 18 views

KV Graft Steering transfers concepts by freezing KV states

A demonstration shows that freezing key-value (KV) states from a guide text allows a Qwen2.5-3B-Instruct model to apply new instructions without including the guide in the live prompt. The method involves running a guide through the model once, freezing its KV cache, and then generating responses on top of that hidden prefix.

arxiv arXiv cs.CL · 21d ago · 23 views

Consolidating 200+ apps onto one self-hosted LLM via multi-axis GRPO and SLERP

The article describes a method for consolidating traffic from over 200 internal applications onto a single self-hosted large language model to address data-residency constraints and GPU fragmentation. The approach closes quality gaps in instruction following, function-calling, and internal task distribution by training separate GRPO experts per axis and merging them via two-stage SLERP.

media MarkTechPost · 25d ago · 52 views

Z.ai and Alibaba independently converge on 3:1 linear attention architecture for GLM-5.3-Flash and Qwen3.8-Flash-Next

Z.ai released GLM-5.3-Flash, a 320B-parameter multimodal MoE model with 18B active parameters, while Alibaba's Qwen team released Qwen3.8-Flash-Next, a 125B model with 6B active parameters. Despite independent development, both teams adopted nearly identical architectural configurations.

media Hugging Face Forums · 31d ago · 28 views

Noah DeAngelis builds experimental routing-based attention for GPT models

Noah DeAngelis has developed an experimental routing-based attention mechanism for GPT models, implemented in a PyTorch project named RoutingGPT. The approach uses a learned linear layer to select a subset of tokens as keys and values while keeping all tokens as queries.