github llama.cpp · just now · inference

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

from English

The llama.cpp project released build b10534, which introduces hardware-specific switch points to tune the crossover between vector and tensor-core paths during quantized decoding on CUDA.

The release adds runtime configuration via GGML_CUDA_MMVQ_MAX to adjust the batch size threshold where the kernel switches from mul_mat_vec_q to MMQ.
Specific switch point values are now included for Blackwell, DGX Spark, and Ada architectures, reducing reliance on environment variables.
Lowering this threshold routes larger batches to the int8 MMQ tensor-core path, yielding a measured 23-41% speedup at batch size 8 on RTX 5090 for Q4_K dense models.
The update includes binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

This change improves inference performance on modern NVIDIA GPUs by automatically selecting the most efficient kernel path based on batch size and hardware capabilities.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

lab Hugging Face Blog · 3d ago · 5 views

Constraint-aware GPU allocator improves utilization and priority-weighted output over FIFO

A constraint-aware GPU allocator was benchmarked against a FIFO scheduler across seven scenarios, achieving up to 33 percentage points higher GPU utilization and up to 105% more priority-weighted output on identical hardware. The system treats real-time inference demand as a fluctuating curve rather than a static reservation, allowing batch-like jobs to fill capacity during troughs while respecting strict constraints like contiguous GPU blocks and non-preemption.

media r/LocalLLaMA · 4d ago · 22 views

NVIDIA serves Qwen 3.8 2.4T on GB300 NVL72 at 4K tokens/s per GPU

NVIDIA has demonstrated serving the Qwen 3.8 2.4T parameter model with configurable reasoning capabilities on its GB300 NVL72 hardware platform.

github llama.cpp · 14d ago · 26 views

llama.cpp b10306 adds SYCL GLU flat path and consolidates kernels

The llama.cpp b10306 release introduces performance optimizations for the SYCL backend, specifically targeting Gated Linear Unit (GLU) operations. The update adds a contiguous fast path for fused GLU ops and consolidates previously distinct kernels to share a common launcher.

github llama.cpp · 17d ago · 31 views

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

media r/LocalLLaMA · 21d ago · 27 views

Lucebox and AMD beat Nvidia DGX Spark by 3.63x on DeepSeek V4 Flash

Lucebox has partnered with AMD to demonstrate a heterogeneous consumer hardware setup that outperforms the Nvidia DGX Spark in inference speed. The system combines an AMD Radeon AI PRO R9700 GPU with a Strix Halo processor to run the full 284B DeepSeek V4 Flash model.