github llama.cpp · 8h ago · inference

llama.cpp b10653 adds TENSOR_READ_LAZY for model loading

from English

The llama.cpp project has released build b10653, introducing the TENSOR_READ_LAZY feature to its model loader. This change includes the addition of the TENSOR_GET_ROW_LAZY function and a corresponding --tensor-read-lazy command-line flag.

Added TENSOR_READ_LAZY and TENSOR_GET_ROW_LAZY to the model loader.
Introduced the --tensor-read-lazy CLI option to enable lazy tensor reading.
Provided binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Windows (CPU, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm), Android, and iOS.

This release updates the core library with new lazy loading capabilities while distributing pre-built binaries across supported platforms.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp adds Qwen3.8-Flash-Next (qwen4exp) architecture support

The llama.cpp project has added support for the Qwen3.8-Flash-Next (qwen4exp) model architecture, implementing its specific components including hyper-connections, gated delta nets, Mixture of Experts, and PLE n-gram hash embeddings.

github llama.cpp · 6d ago · 11 views

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2

The llama.cpp project released build b10568, which updates the model handling to use the `ggml_rope_set_offset()` function. This change is partially applied to support DeepSeek 2.

github llama.cpp · 7d ago · 17 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 7d ago · 7 views

llama.cpp b10532 adds Metal KV dequantization for Flash Attention

The llama.cpp project released version b10532, introducing a preprocessing pass on the Metal backend that dequantizes quantized Key-Value (KV) caches to F16 before executing Flash Attention.

github llama.cpp · 7d ago · 13 views

llama.cpp b10517 adds Vulkan FA dequant optimizations and fixes

The llama.cpp project released build b10517, introducing several Vulkan-specific improvements to the flash attention (FA) dequantization path. Key changes include dequantizing q8_0 KV once in coopmat1, skipping the FA dequant path on coopmat2, and excluding Intel Xe1 from the FA dequant path.