github llama.cpp · 1h ago · inference

llama.cpp b10080 fixes server crash and UI validation error

from English

The llama.cpp b10080 release addresses two specific bugs in the server and user interface components. The server now correctly returns a 400 status code instead of crashing with a 500 error when validation fails for requests containing the X-Conversation-Id header. Additionally, the UI no longer sends the backend_sampling placeholder, which previously caused default settings completions to fail validation.

Server: Fixes a std::bad_function_call crash by handling empty next_orig in on_complete() when set_next() is skipped during validation errors.
Server: Evicts sessions installed by set_req() for failed requests to prevent resource leaks and discovery issues.
UI: Guards the backend_sampling field with hasValue() to filter out empty string placeholders while preserving explicit false values.

These changes improve stability by preventing server crashes on invalid input and ensuring that default settings are correctly transmitted from the client.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10081 resolves draft repo to its requested sidecar

The llama.cpp b10081 release fixes a bug where the `--hfd` flag pointed to a repository containing speculative sidecars, but the draft plan incorrectly resolved to the main model instead of the sidecar. This occurred because `find_best_model()` excluded sidecar files and the explicit draft plan suppressed sidecar discovery.

arxiv arXiv cs.LG · 5h ago

SkewAdam uses tiered optimizer state to reduce MoE training memory by 97%

Researchers introduce SkewAdam, an optimizer designed for mixture-of-experts (MoE) models that allocates different state types to distinct parameter populations to drastically reduce memory usage. By assigning float32 momentum and factored second moments to the backbone, factored second moments to experts, and exact second moments to the router, SkewAdam reduces optimizer state from 50.6 GB to 1.29 GB for a 6.78B-parameter model.

github llama.cpp · 7h ago

llama.cpp b10078 refactors Vulkan queue handling for host-side mutex bypass

The llama.cpp project released build b10078, which includes a significant refactor of the Vulkan backend to improve synchronization efficiency. The update integrates support for VK_KHR_internally_synchronized_queues, allowing the library to abstract queue submission into a polymorphic interface that completely bypasses host-side mutex locking when the driver supports it.

github llama.cpp · 14h ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

media Hugging Face Forums · 2d ago · 1 view

Production voice agents with local LLMs face persistent latency and context limits

A team running two production voice agents in Spain using Retell AI, Ollama, and local models on an RTX 4000 Ada reports that while conversation quality improved, the core latency problem remains unsolved. The primary constraint is a ~3-second timeout from Retell AI; if the LLM fails to produce a first token within this window, the connection drops, leading to call termination after two reconnections.