github llama.cpp · just now · inference

llama.cpp b10249 adds get_info tool to server

from English

The llama.cpp project released version b10249, which introduces a new `get_info` tool to the server component. This update also includes documentation fixes for the `--rpc` flag.

The server now exposes a `get_info` tool for retrieving system information.
Handling of the probe result was hardened to report 'unknown' OS when the process fails or times out, preventing diagnostic text from being misinterpreted as an OS name.
Output stripping and timeout limits were added to ensure clean data handling.
The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, OpenVINO, SYCL, HIP), and openEuler.

This release makes the `get_info` tool available for server-side diagnostics while providing updated binaries across all supported platforms.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 2h ago

llama.cpp b10247 fixes crashes with wide MoE models by using dynamic allocation

The llama.cpp project released version b10247, which addresses stability issues when loading large Mixture of Experts (MoE) models on multi-backend setups. The update replaces fixed-size arrays in the backend scheduler with dynamically allocated buffers to handle graph splits exceeding 30 input tensors.

github llama.cpp · 3h ago

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.

github llama.cpp · 8h ago · 2 views

llama.cpp b10242 adds CUDA backend sampler for penalties

The llama.cpp project released version b10242, introducing a new backend sampler for penalties that allows penalty handling to be offloaded to the GPU. This change enhances the `common_sampler_init` function and updates the `llama_sampler_penalties` structure to inherit from `llama_sampler_backend`, enabling frequency and presence adjustments on the backend.

github llama.cpp · 11h ago · 1 view

llama.cpp b10241 fixes CUDA SMEM data-races in block_reduce

The llama.cpp project released version b10241, which addresses a critical bug in the CUDA backend. The update resolves data-races that occurred when reusing shared memory (SMEM) during block reductions.

media Hugging Face Forums · 14h ago · 1 view

Quesma publishes curated map of 218 tools and papers for LLM token cost efficiency

Quesma has released an open, curated list containing 218 entries that track where tokens are spent in LLM and agent workflows. The resource covers monitoring, semantic caching, cheap local models, context engineering, and cost governance.