github llama.cpp · 1h ago · inference

llama.cpp b10926 release handles unsupported tq1_0 quants gracefully

from English

The llama.cpp project has released version b10926, which includes a fix to handle unsupported tq1_0 quantizations by failing gracefully. This update addresses issue #28681 to improve stability when encountering this specific quantization format.

The release provides binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu with CPU, Vulkan, ROCm 10.0, OpenVINO, and SYCL backends), Android (arm64), Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, and ROCm 10.0), and openEuler architectures.

This update ensures that users can continue using llama.cpp without crashes when models utilize the tq1_0 quantization scheme.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10921 aligns WebGPU tensor bindings to block size

The llama.cpp project has released build b10921, which includes a fix for WebGPU by aligning tensor bindings to the type block size. This change ensures that block quantized views receive a valid element offset in the shader by walking the binding offset back until the distance to the tensor is a whole number of blocks.

github llama.cpp · 1d ago · 1 view

llama.cpp b10906 fixes speculation after an image in the server

The llama.cpp project released version b10906, which includes a fix for the server component regarding speculative decoding after image inputs. The change ensures that the actual position is passed to the drafter instead of the token count, affecting all drafters rather than just DFlash.

github llama.cpp · 2d ago · 3 views

llama.cpp b10899 adds Vulkan optimizations for Qwen and split_k support

The llama.cpp project released version b10899, introducing specific performance improvements to its Vulkan backend. The update focuses on optimizing matrix operations for the Qwen model family and enhancing general small M performance.

github llama.cpp · 2d ago · 4 views

llama.cpp b10896 fixes DFlash decoding failure with vision models

The llama.cpp project released build b10896, which addresses a critical issue where the drafter memory failed to allocate new tokens when using DFlash with vision models. This failure occurred because images reported a fixed offset, preventing proper token allocation.

github llama.cpp · 4d ago · 2 views

llama.cpp disables lazy tensor loading by default on iGPUs

The llama.cpp project has disabled lazy tensor loading by default for integrated GPUs (iGPUs) in release b10867. This change addresses a regression introduced in the previous "AUTO" lazy mode setting.