github llama.cpp · 1h ago · inference

llama.cpp b10327 fixes CUDA thread/block count in quantized cpy kernel

from English

The llama.cpp project released version b10327, which includes a fix for the thread and block count in quantized copy kernel launches on CUDA. This update addresses an issue identified in pull request #26731.

The release adds tests for uneven block count copy cases to ensure correctness.
Binaries are available for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Android, Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.
An iOS XCFramework and a standalone UI build are also provided.

This release ensures stable execution of quantized copy operations on CUDA hardware across a wide range of supported platforms.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 13h ago

llama.cpp b10313 adds LRU scheduler to the server

The llama.cpp project has released version b10313, which introduces a Least Recently Used (LRU) scheduler for its server component. This update includes handling for request coalescing and fixes for streaming cases.

github llama.cpp · 15h ago

llama.cpp b10312 release with server fix for busy models

The llama.cpp project has released version b10312, which includes a specific update to the server component. The key change addresses an issue where busy models were being evicted from the router.

github llama.cpp · 17h ago

llama.cpp b10307 fixes SYCL NVFP4 parsing

The llama.cpp project released version b10307, which includes a fix for the SYCL GPU code to correctly parse UE4M3 scaling factors used in NVFP4 quantization.

github llama.cpp · 1d ago

llama.cpp b10298 adds mtmd chunk save/load function

The llama.cpp project has released build b10298, which introduces a new chunk save and load function for the mtmd module. This update also includes code cleanup and additional tests.

github llama.cpp · 3d ago · 1 view

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

The llama.cpp project has released version b10282, which introduces new monitoring capabilities for speculative decoding. The server now includes spec-decode counters in its /metrics endpoint, allowing users to track performance metrics related to this feature.