arxiv arXiv cs.LG · 55d ago · research

StreamKL: Fast and Memory-Efficient KL Divergence for Attention Distillation

from English

StreamKL introduces a fused GPU primitive that eliminates quadratic memory usage in attention distillation by streaming query-key tiles through on-chip SRAM. It achieves up to 43x speedup in forward and 14x in backward passes, reducing extra HBM footprint from O(N_QN_K) to O(1), enabling long-context distillation on a single GPU.

Importance 3/3 New feature vs. leaders New harness with differentiators arXiv cs.LG NVIDIA Evaluation & benchmarks Inference efficiency Training methods

Read original

media r/LocalLLaMA · 54d ago · 3 views

Fixing Long-Context Decode Cliff on Radeon R9700 with vLLM 0.22.1

A long-context decode performance cliff on AMD Radeon AI PRO R9700 (RDNA4) was resolved by enabling AITER Unified Attention in vLLM 0.22.1. The fix involves relaxing a CDNA gate to include RDNA4, disabling other attention backends, and using bf16 KV cache, resulting in significant speedups across all context lengths. FP8 KV is ineffective on this hardware, and the model's native 262K context is fully achievable with bf16, offering ~2.9× concurrency without needing FP8.

arxiv arXiv cs.LG · 56d ago

FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers

FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.

arxiv arXiv cs.AI · 56d ago

FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers

FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over efficient baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.

lab NVIDIA Technical Blog · 33d ago

NVIDIA explains host offloading to reduce HBM bottlenecks in JAX LLM training

Large language model (LLM) training workloads increasingly hit GPU memory limits before compute is fully utilized. Model weights, gradients, optimizer states, communication buffers, and intermediate activations all compete for limited GPU high-bandwidth memory (HBM). As model size, sequence length, and batch size grow, HBM capacity often becomes the primary scaling bottleneck.

lab Hugging Face Blog · 34d ago

PyTorch profiling shows in-place masking removes memory copy and SDPA dispatches to optimized backends

This article demonstrates how to profile attention mechanisms in PyTorch using the profiler traces to identify performance bottlenecks and optimization opportunities. It compares naive attention implementations against PyTorch's built-in Scaled Dot Product Attention (SDPA) to illustrate kernel behavior.