github llama.cpp · 2h ago · inference

llama.cpp b10078 refactors Vulkan queue handling for host-side mutex bypass

from English

The llama.cpp project released build b10078, which includes a significant refactor of the Vulkan backend to improve synchronization efficiency. The update integrates support for VK_KHR_internally_synchronized_queues, allowing the library to abstract queue submission into a polymorphic interface that completely bypasses host-side mutex locking when the driver supports it.

Refactors vk_queue to use per-instance mutexes and unique handles for exclusive ownership.
Integrates VK_KHR_internally_synchronized_queues to bypass host-side mutex locking when supported by the driver.
Adds fallback defines and static constexpr constants for internallySynchronizedQueues detection.
Fixes compilation errors, duplicate pNext chains, and null checks in the vk_device_struct destructor.

This change optimizes lifetime management and thread safety while reducing redundant API calls during device creation.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 8h ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

media Hugging Face Forums · 2d ago · 1 view

Production voice agents with local LLMs face persistent latency and context limits

A team running two production voice agents in Spain using Retell AI, Ollama, and local models on an RTX 4000 Ada reports that while conversation quality improved, the core latency problem remains unsolved. The primary constraint is a ~3-second timeout from Retell AI; if the LLM fails to produce a first token within this window, the connection drops, leading to call termination after two reconnections.

github llama.cpp · 2d ago · 1 view

llama.cpp b10069 adds OpenCL broadcast support for Adreno MUL_MAT

The llama.cpp project released build b10069, which includes specific OpenCL updates for Qualcomm Adreno GPUs. The primary change involves supporting broadcast operations for the Adreno MUL_MAT kernel and honoring view offsets for the Q8_0 MUL_MAT variant to enable multi-stream functionality in llama-server.

media Hugging Face Forums · 3d ago · 2 views

zhongkaifu releases TensorSharp, a native .NET LLM inference engine for GGUF models

zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

github llama.cpp · 3d ago · 1 view

llama.cpp b10068 rotates injected K/V cache for DFlash

The llama.cpp project released version b10068, which introduces a change to the DFlash model implementation. The update modifies src/models/dflash.cpp to rotate the injected key/value (K/V) cache when K/V quantization is used.