github llama.cpp · 1h ago · inference

llama.cpp b10312 release with server fix for busy models

from English

The llama.cpp project has released version b10312, which includes a specific update to the server component. The key change addresses an issue where busy models were being evicted from the router.

Server: Prevents eviction of busy models (PR #26567).
macOS/iOS: Provides binaries for Apple Silicon (arm64) and Intel (x64), along with an iOS XCFramework. KleidiAI support on macOS is disabled.
Linux: Offers builds for Ubuntu x64, arm64, and s390x CPUs, as well as Vulkan, ROCm 7.2, OpenVINO, SYCL FP32, and SYCL FP16 variants.
Android: Includes an arm64 CPU build.
Windows: Supports x64 and arm64 CPUs, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, and HIP.
openEuler: Builds for x86 (310p, 910b ACL Graph) and aarch64 (310p, 910b ACL Graph) are available, while the standard pull request is disabled.

This release ensures continuous availability of active models in server deployments and provides comprehensive binaries across major operating systems and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10313 adds LRU scheduler to the server

The llama.cpp project has released version b10313, which introduces a Least Recently Used (LRU) scheduler for its server component. This update includes handling for request coalescing and fixes for streaming cases.

github llama.cpp · 4h ago

llama.cpp b10307 fixes SYCL NVFP4 parsing

The llama.cpp project released version b10307, which includes a fix for the SYCL GPU code to correctly parse UE4M3 scaling factors used in NVFP4 quantization.

github llama.cpp · 19h ago

llama.cpp b10298 adds mtmd chunk save/load function

The llama.cpp project has released build b10298, which introduces a new chunk save and load function for the mtmd module. This update also includes code cleanup and additional tests.

github llama.cpp · 2d ago · 1 view

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

The llama.cpp project has released version b10282, which introduces new monitoring capabilities for speculative decoding. The server now includes spec-decode counters in its /metrics endpoint, allowing users to track performance metrics related to this feature.

github llama.cpp · 3d ago

llama.cpp b10259 allows tensor reshape during load

The llama.cpp project has released build b10259, which introduces the ability to allow the reshape of tensors during the loading process.