github llama.cpp · 1d ago · inference

llama.cpp b10067 excludes DeepSeek-V4 routing table from quantization

from English

The llama.cpp project released version b10067, which includes a fix for the `llama-quantize` tool to correctly handle DeepSeek-V4 models. The update prevents the quantizer from attempting to convert the `ffn_gate_tid2eid` tensor, an i32 token-to-expert index table, into a float type.

Excludes the `ffn_gate_tid2eid` routing table from quantization to avoid conversion failures.
Adds the tensor to the name-based exclusion list alongside `ffn_gate_inp.weight`.
Resolves issue #25754 where quantization previously failed on DeepSeek-V4 models.

This change allows users to successfully quantize DeepSeek-V4 models without encountering errors related to integer-to-float type conversion.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 1d ago

llama.cpp b10068 rotates injected K/V cache for DFlash

The llama.cpp project released version b10068, which introduces a change to the DFlash model implementation. The update modifies src/models/dflash.cpp to rotate the injected key/value (K/V) cache when K/V quantization is used.

github llama.cpp · 2d ago

llama.cpp b10066 adds OpenCL kernel_gemm_moe_q6_k_f32_ns support

The llama.cpp project has released version b10066, introducing support for the `kernel_gemm_moe_q6_k_f32_ns` kernel via OpenCL. This update enables the loading and usage of this specific binary kernel library within the framework.

github llama.cpp · 2d ago

llama.cpp b10064 release with OpenCL transpose optimization

The llama.cpp project has released version b10064, which includes a change to the OpenCL backend. This update modifies the q4_K quantization format by transposing noshuffle scales to enable coalesced reads.

github llama.cpp · 2d ago

llama.cpp b10059 sets ggml-blas default hadamard mul_mat to cpu routine

The llama.cpp project has released version b10059, which updates the ggml-blas library to use the CPU routine as the default for hadamard matrix multiplication.

github llama.cpp · 2d ago

llama.cpp b10057 fixes SYCL Q5_K kernels and releases binaries

The llama.cpp project has released version b10057, which includes several bug fixes for the SYCL backend alongside standard binary distributions across multiple platforms.