github llama.cpp · 1h ago · inference

llama.cpp b10519 refactors sleep handling to allow metrics access

from English

The llama.cpp project released version b10519, which includes a significant refactor of the server's sleep handling mechanism. This change allows the /metrics endpoint to remain accessible while the server is in a sleeping state.

Refactored on_sleeping_state logic and split metrics and slots tasks/results.
Added cached responses and fixed a race condition in the metrics task timer.
Fixed get_res_model_info and updated documentation.
Provided binaries for macOS (Apple Silicon/Intel), Linux (CPU, Vulkan, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, ROCm), Android, and openEuler.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 4h ago · 1 view

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.

media r/LocalLLaMA · 9h ago · 2 views

G9v3-39A5B MoE model shows strong results on artificialanalysis

The G9v3-39A5B model is drawing attention for its performance on artificialanalysis, where it reportedly ranks above Qwen 3.6 27B.

media Hugging Face Forums · 9h ago · 1 view

Celestium pipeline enables SDXL on 4GB-8GB GPUs via adaptive FP16 and CPU fallback

A diagnostic using Celestium’s hardware-agnostic inference pipeline demonstrates that Stable Diffusion XL can run stably on low-VRAM GPUs (4GB, 6GB, and 8GB) by leveraging adaptive FP16 precision and autonomous memory cleanup. The approach prevents fragmentation spikes during multi-stage pipelines and utilizes an emergency CPU mode to maintain generation when GPU VRAM is exhausted.

github llama.cpp · 19h ago · 1 view

llama.cpp b10505 adds dedup-cache-models preset option

The llama.cpp project has released version b10505, which introduces a new `dedup-cache-models` preset option for the server component.

media Hugging Face Forums · 2d ago · 3 views

Fourier Magnitude KV Cache Quantization doubles context length at same memory

ntrillard demonstrates that quantizing the Fourier magnitude spectrum of K and V cache values to 4-bit while preserving phase at full precision achieves 96.9% token match with fp16 reference on Gemma-3-1B, a 5-10x improvement over standard min-max quantization.