github llama.cpp · 1h ago · inference

llama.cpp b10313 adds LRU scheduler to the server

from English

The llama.cpp project has released version b10313, which introduces a Least Recently Used (LRU) scheduler for its server component. This update includes handling for request coalescing and fixes for streaming cases.

The server now supports an LRU scheduler to manage connections.
Request coalescing logic is implemented to handle waiting queue exits.
Tests have been added to verify the new scheduling behavior.
A bug affecting the stream case has been fixed.

This release provides updated binaries for macOS, Linux, Windows, Android, and iOS across various hardware backends including CPU, CUDA, Vulkan, ROCm, and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 2h ago

llama.cpp b10312 release with server fix for busy models

The llama.cpp project has released version b10312, which includes a specific update to the server component. The key change addresses an issue where busy models were being evicted from the router.

github llama.cpp · 5h ago

llama.cpp b10307 fixes SYCL NVFP4 parsing

The llama.cpp project released version b10307, which includes a fix for the SYCL GPU code to correctly parse UE4M3 scaling factors used in NVFP4 quantization.

github llama.cpp · 20h ago

llama.cpp b10298 adds mtmd chunk save/load function

The llama.cpp project has released build b10298, which introduces a new chunk save and load function for the mtmd module. This update also includes code cleanup and additional tests.

github llama.cpp · 2d ago · 1 view

llama.cpp b10282 adds spec-decode counters to /metrics endpoint

The llama.cpp project has released version b10282, which introduces new monitoring capabilities for speculative decoding. The server now includes spec-decode counters in its /metrics endpoint, allowing users to track performance metrics related to this feature.

github llama.cpp · 3d ago

llama.cpp b10259 allows tensor reshape during load

The llama.cpp project has released build b10259, which introduces the ability to allow the reshape of tensors during the loading process.