github llama.cpp · 1h ago · inference

llama.cpp b10081 resolves draft repo to its requested sidecar

from English

The llama.cpp b10081 release fixes a bug where the `--hfd` flag pointed to a repository containing speculative sidecars, but the draft plan incorrectly resolved to the main model instead of the sidecar. This occurred because `find_best_model()` excluded sidecar files and the explicit draft plan suppressed sidecar discovery.

The fix wires the discovered sidecars as the draft, following the fallback pattern of the main plan, allowing commands like `llama-server -hf repo -hfd repo --spec-type draft-dflash` to work as expected.

This release includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10080 fixes server crash and UI validation error

The llama.cpp b10080 release addresses two specific bugs in the server and user interface components. The server now correctly returns a 400 status code instead of crashing with a 500 error when validation fails for requests containing the X-Conversation-Id header. Additionally, the UI no longer sends the backend_sampling placeholder, which previously caused default settings completions to fail validation.

arxiv arXiv cs.LG · 6h ago

SkewAdam uses tiered optimizer state to reduce MoE training memory by 97%

Researchers introduce SkewAdam, an optimizer designed for mixture-of-experts (MoE) models that allocates different state types to distinct parameter populations to drastically reduce memory usage. By assigning float32 momentum and factored second moments to the backbone, factored second moments to experts, and exact second moments to the router, SkewAdam reduces optimizer state from 50.6 GB to 1.29 GB for a 6.78B-parameter model.

github llama.cpp · 8h ago

llama.cpp b10078 refactors Vulkan queue handling for host-side mutex bypass

The llama.cpp project released build b10078, which includes a significant refactor of the Vulkan backend to improve synchronization efficiency. The update integrates support for VK_KHR_internally_synchronized_queues, allowing the library to abstract queue submission into a polymorphic interface that completely bypasses host-side mutex locking when the driver supports it.

github llama.cpp · 15h ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

media Hugging Face Forums · 2d ago · 1 view

Production voice agents with local LLMs face persistent latency and context limits

A team running two production voice agents in Spain using Retell AI, Ollama, and local models on an RTX 4000 Ada reports that while conversation quality improved, the core latency problem remains unsolved. The primary constraint is a ~3-second timeout from Retell AI; if the LLM fails to produce a first token within this window, the connection drops, leading to call termination after two reconnections.