github llama.cpp · 3h ago · inference

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

from English

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

The implementation is gated at compile time and runtime, requiring 16-byte alignment and specific stride conditions to ensure correctness. An occupancy gate prevents regression on small single-row gathers by keeping them on the scalar path. On Strix Halo (gfx1151), this optimization reduced DeltaNet recurrent-state gather latency from 18.6us to 13.0us.

The release includes binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, Vulkan, OpenVINO, SYCL, and OpenCL backends.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

media Hugging Face Forums · 3d ago · 2 views

zhongkaifu releases TensorSharp, a native .NET LLM inference engine for GGUF models

zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

media Hugging Face Forums · 1d ago · 1 view

Production voice agents with local LLMs face persistent latency and context limits

A team running two production voice agents in Spain using Retell AI, Ollama, and local models on an RTX 4000 Ada reports that while conversation quality improved, the core latency problem remains unsolved. The primary constraint is a ~3-second timeout from Retell AI; if the LLM fails to produce a first token within this window, the connection drops, leading to call termination after two reconnections.

github llama.cpp · 2d ago · 1 view

llama.cpp b10069 adds OpenCL broadcast support for Adreno MUL_MAT

The llama.cpp project released build b10069, which includes specific OpenCL updates for Qualcomm Adreno GPUs. The primary change involves supporting broadcast operations for the Adreno MUL_MAT kernel and honoring view offsets for the Q8_0 MUL_MAT variant to enable multi-stream functionality in llama-server.

github llama.cpp · 3d ago · 1 view

llama.cpp b10068 rotates injected K/V cache for DFlash

The llama.cpp project released version b10068, which introduces a change to the DFlash model implementation. The update modifies src/models/dflash.cpp to rotate the injected key/value (K/V) cache when K/V quantization is used.

github llama.cpp · 3d ago · 1 view

llama.cpp b10067 excludes DeepSeek-V4 routing table from quantization

The llama.cpp project released version b10067, which includes a fix for the `llama-quantize` tool to correctly handle DeepSeek-V4 models. The update prevents the quantizer from attempting to convert the `ffn_gate_tid2eid` tensor, an i32 token-to-expert index table, into a float type.