github llama.cpp · 1h ago · inference

llama.cpp b11005 accepts zeroed rope probe to reduce graph splits

from English

The llama.cpp project released version b11005, which includes a fix for the Hexagon backend to accept zeroed rope probes in supports_op.

Previously, rejecting probes with n_dims == 0 or freq_base == 0 forced rope_freqs onto the CPU, causing unnecessary graph splits. This change prevents such splits, reducing the number of divisions in models like gemma-4-E2B from five to two.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 2 views

llama.cpp adds support for HrmTextForCausalLM (DFM Mimir 1B)

The llama.cpp project has added support for the HrmTextForCausalLM architecture, specifically enabling inference for the DFM Mimir 1B model. This implementation introduces a new GGUF writer and loader to handle the model's unique structure, which runs two transformer stacks in an alternating cycle over the same token stream.

github llama.cpp · 3h ago · 1 view

llama.cpp b11001 fixes int16 transpose corruption on RISC-V

The llama.cpp project released build b11001, which includes a critical fix for the `permute_transpose_impl` function handling int16 data on RISC-V architectures.

github llama.cpp · 10h ago · 1 view

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

github llama.cpp · 18h ago · 2 views

llama.cpp b10989 enables AllReduce for ROCm and provides multi-platform binaries

The llama.cpp project has released version b10989, which introduces support for enabling AllReduce operations within the ROCm backend. This update is part of the ongoing development to expand hardware compatibility and distributed computing capabilities for large language model inference.

github llama.cpp · 23h ago · 1 view

llama.cpp b10988 adds OpenCL MoE expert matmul selection for speculative decoding

The llama.cpp project released build b10988, introducing changes to the OpenCL backend specifically for Mixture of Experts (MoE) models. The update implements logic to choose the MoE expert matrix multiplication based on batch size during speculative decoding and multi-token prediction.