A long-context decode performance cliff on AMD Radeon AI PRO R9700 (RDNA4) was resolved by enabling AITER Unified Attention in vLLM 0.22.1. The fix involves relaxing a CDNA gate to include RDNA4, disabling other attention backends, and using bf16 KV cache, resulting in significant speedups across all context lengths. FP8 KV is ineffective on this hardware, and the model's native 262K context is fully achievable with bf16, offering ~2.9× concurrency without needing FP8.
media
r/LocalLLaMA
·
50d ago
·
open_models
Fixing Long-Context Decode Cliff on Radeon R9700 with vLLM 0.22.1
from English
Related articles
arxiv
arXiv cs.AI
·
51d ago
·
2 views
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
UltraQuant enables 4-bit KV caching for context-heavy agents, reducing P50 time-to-first-token by 3.47x in late rounds and boosting output throughput by 1.63x over FP8 KV baseline. It achieves this using FP8 queries, FP4 KV tensors, UE8M0 group scales, and native scaled-MFMA on AMD CDNA4 GPUs, with optimizations for decode-attention kernels and robust design choices like asymmetric K/V treatment and Walsh-Hadamard rotation.
arxiv
arXiv cs.LG
·
51d ago
·
1 view
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
UltraQuant introduces a 4-bit KV caching method tailored for context-heavy agent workloads. It achieves 3.47x reduction in P50 time-to-first-token in late rounds and 1.63x higher output throughput compared to FP8 KV caching, using FP8 queries, FP4 KV tensors, and native AMD CDNA4 scaled-MFMA support.
arxiv
arXiv cs.LG
·
52d ago
FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers
FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.
arxiv
arXiv cs.AI
·
52d ago
FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers
FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over efficient baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.
lab
Hugging Face Blog
·
30d ago
PyTorch profiling shows in-place masking removes memory copy and SDPA dispatches to optimized backends
This article demonstrates how to profile attention mechanisms in PyTorch using the profiler traces to identify performance bottlenecks and optimization opportunities. It compares naive attention implementations against PyTorch's built-in Scaled Dot Product Attention (SDPA) to illustrate kernel behavior.