github llama.cpp · 1h ago · inference

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

from English

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

Adds k-quant support (q2_K to q6_K) to handle common GGUF recipes like Q4_K_M.
Extends shared super-block dequantizers to nine i-quants with a 32-thread layout.
Moves mxfp4 dequantizer into shared helpers, closing GET_ROWS type coverage on CUDA.
Gates row size checks only for 32-value sub-block types (iq4_nl and mxfp4) to prevent unnecessary host fallbacks.

By preventing the scheduler from falling back to the host for these quantization types, the update avoids copying the full embedding matrix back on every token in single-device graphs.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 12h ago

llama.cpp b10085 fixes Qwen3-VL grounding by aligning position embedding interpolation

The llama.cpp project released build b10085, which includes a fix for the Qwen3-VL vision model's position embedding interpolation to resolve grounding coordinate mismatches.

github llama.cpp · 1d ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

github llama.cpp · 3d ago · 1 view

llama.cpp b10069 adds OpenCL broadcast support for Adreno MUL_MAT

The llama.cpp project released build b10069, which includes specific OpenCL updates for Qualcomm Adreno GPUs. The primary change involves supporting broadcast operations for the Adreno MUL_MAT kernel and honoring view offsets for the Q8_0 MUL_MAT variant to enable multi-stream functionality in llama-server.

media Hugging Face Forums · 4d ago · 2 views

zhongkaifu releases TensorSharp, a native .NET LLM inference engine for GGUF models

zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

github llama.cpp · 4d ago · 1 view

llama.cpp b10068 rotates injected K/V cache for DFlash

The llama.cpp project released version b10068, which introduces a change to the DFlash model implementation. The update modifies src/models/dflash.cpp to rotate the injected key/value (K/V) cache when K/V quantization is used.