github llama.cpp · 3h ago · inference

llama.cpp b10656 caps working memory to avoid loading big tensors onto RAM

from English

The llama.cpp project has released build b10656, which includes a change to cap the working memory size. This modification is designed to prevent large tensors from being loaded into system RAM.

The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64, arm64, s390x), Android, Windows, and openEuler.
Supported backends include CPU, Vulkan, ROCm 7.14, OpenVINO, SYCL, CUDA 12/13, and OpenCL Adreno.
KleidiAI support for macOS Apple Silicon is disabled in this build.

This update ensures more stable memory usage during quantization processes by limiting the working memory footprint.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b10662 adds ctx-per-slot for unified KV cache

The llama.cpp project released build b10662, introducing a new `--kv-unified-per-slot` argument that allows users to specify context per slot for the unified KV cache. This change replaces the previous context fraction mechanism with context pool slots and refactors the underlying implementation.

github llama.cpp · 1h ago · 1 view Live

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.

github llama.cpp · 2h ago · 1 view

llama.cpp b10658 adds DFlash2 support and fixes mrope bug

The llama.cpp project released build b10658, introducing support for the DFlash2 specification which includes local convolution and a candidate selector. This update also addresses a specific mrope bug within the codebase.

github llama.cpp · 2h ago · 1 view

llama.cpp bundles HIP runtime DLLs with Windows ROCm release

The llama.cpp project has updated its Windows ROCm release to bundle the necessary HIP runtime DLLs directly alongside the binaries. This change addresses issue #26929 by ensuring the correct HIP runtime loads instead of potentially conflicting with the driver's copy in System32.

github llama.cpp · 3h ago · 1 view

llama.cpp b10657 adds OpenCL bin kernels for GEMO MOE

The llama.cpp project released version b10657, introducing new OpenCL binary kernels for mixed-expert models. Specifically, the update adds `kernel_gemm_moe_q4_0_q8_1_dp4a_bin` and `kernel_gemm_moe_mxfp4_q8_1_dp4a_bin` to support efficient computation on compatible hardware.