arxiv arXiv cs.LG · 2h ago · research

SkewAdam uses tiered optimizer state to reduce MoE training memory by 97%

from English

Researchers introduce SkewAdam, an optimizer designed for mixture-of-experts (MoE) models that allocates different state types to distinct parameter populations to drastically reduce memory usage. By assigning float32 momentum and factored second moments to the backbone, factored second moments to experts, and exact second moments to the router, SkewAdam reduces optimizer state from 50.6 GB to 1.29 GB for a 6.78B-parameter model.

Peak training memory drops from 81.4 GB to 31.3 GB, fitting within a 40 GB accelerator budget.
SkewAdam achieves a validation perplexity of 108.4 over 82M tokens, outperforming AdamW (126.8), Muon (120.2), and Lion (393.7).
The memory savings are achieved without accuracy loss; the performance gain stems from retaining momentum rather than the tiered allocation itself.

The results suggest that where optimizer state is allocated matters as much as its volume, enabling efficient training on hardware with limited memory.

Importance 2/3 arXiv cs.LG Inference efficiency Training methods

Read original

media Hugging Face Forums · 6d ago

OLMo-core ablation shows Engram indexing fixes narrow architecture gap

An ablation study in OLMo-core compared integrating Engram onto a standard Transformer versus a 3 GDN-layer + 1 attention-layer hybrid backbone. The analysis revealed that fixing signed overflow and padded token ID indexing failures substantially narrowed the performance gap between the two architectures.

media r/LocalLLaMA · 9d ago · 1 view

Flint compresses reasoning traces to reduce token usage while maintaining accuracy

A study demonstrates that section-aware compression of self-distilled reasoning traces allows Qwen3.5-4B and Gemma-4-12b models to match or exceed original performance using significantly fewer tokens. By preserving computation and verification spans while compressing narration, the method avoids the greedy decoding failures associated with flat compression.

media r/LocalLLaMA · 9d ago · 1 view

Flash-MSA accelerates million-token training with sparse attention kernels

Flash-MSA is a method designed to accelerate the training of models on sequences containing millions of tokens by utilizing sparse attention kernels.

lab NVIDIA Technical Blog · 11d ago

NVIDIA explains host offloading to reduce HBM bottlenecks in JAX LLM training

Large language model (LLM) training workloads increasingly hit GPU memory limits before compute is fully utilized. Model weights, gradients, optimizer states, communication buffers, and intermediate activations all compete for limited GPU high-bandwidth memory (HBM). As model size, sequence length, and batch size grow, HBM capacity often becomes the primary scaling bottleneck.

arxiv arXiv cs.AI · 12d ago

SLORR introduces simple, stateless in-training low-rank regularization

Researchers introduce SLORR, a framework for in-training low-rank regularization that is stateless and architecture-preserving. It uses GPU-friendly approximations to regularize weight matrices without modifying the model or relying on cached quantities.