github llama.cpp · 3d ago · inference

llama.cpp b10844 adds Vulkan DeepSeek-V4 hyper-connection fused ops

from English

The llama.cpp project released version b10844, introducing support for DeepSeek-V4 hyper-connection fused operations (DSV4_HC_COMB/PRE/POST) in the Vulkan backend. This update brings Vulkan to parity with CUDA and Metal regarding these specific optimizations.

The new dsv4_hc_comb operation runs the full 20-iteration Sinkhorn algorithm in registers, replacing approximately 137 ordered node executions per site with a single dispatch.
dsv4_hc_pre and dsv4_hc_post handle elementwise stream collapse and fan-out using shared memory for per-token coefficients.
The unfused Sinkhorn comb chain previously accounted for about 32% of decode op time on gfx1151 hardware across roughly 16k dispatches per token.
Environment variables GGML_VK_DISABLE_DSV4_HC, along with individual _COMB, _PRE, and _POST flags, allow users to disable these operations independently.

This change eliminates the performance bottleneck caused by the unfused primitive chain for Vulkan users running DeepSeek-V4 models.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 2d ago · 2 views

llama.cpp disables lazy tensor loading by default on iGPUs

The llama.cpp project has disabled lazy tensor loading by default for integrated GPUs (iGPUs) in release b10867. This change addresses a regression introduced in the previous "AUTO" lazy mode setting.

github llama.cpp · 2d ago · 3 views

llama.cpp b10864 fixes checkpoint eviction for hybrid models

The llama.cpp project released build b10864, which includes a fix for the checkpoint spacing eviction logic in `create_checkpoint()`. Previously, the eviction rule kept the oldest checkpoint and erased later ones within `checkpoint_min_step`, causing prompts shorter than that threshold to drop the checkpoint needed for resumption. This forced hybrid and recurrent models to unnecessarily re-prefill from the previous checkpoint.

github llama.cpp · 3d ago · 4 views

llama.cpp b10842 adds gfx90c HIP support

The llama.cpp project released build b10842, which introduces support for the gfx90c architecture via HIP.

media Hugging Face Forums · 3d ago · 8 views

AI Model Radar launches with measured hardware fit for open models

A new site called AI Model Radar helps users determine which open models actually fit their GPU or Mac by using measured data rather than estimates. The platform calculates weight sizes based on real published GGUF files and computes context cache layer-by-layer from model configurations.

github llama.cpp · 3d ago · 7 views

llama.cpp b10840 adds branchless Q4_K/Q5_K unpack and L2 prefetch for CUDA

The llama.cpp project released version b10840, which introduces performance optimizations for CUDA inference. The update implements branchless computation for Q4_K and Q5_K quantization formats to accelerate matrix-vector multiplication (mmvq).