github llama.cpp · just now · inference

llama.cpp adds support for HrmTextForCausalLM (DFM Mimir 1B)

from English

The llama.cpp project has added support for the HrmTextForCausalLM architecture, specifically enabling inference for the DFM Mimir 1B model. This implementation introduces a new GGUF writer and loader to handle the model's unique structure, which runs two transformer stacks in an alternating cycle over the same token stream.

The conversion process maps fused gqkv projections to llama.cpp q/k/v plus a separate sigmoid gate tensor.
The KV cache handles block aliasing for looped architectures, holding one entry per pass across 128 layers.
Inference is verified against Hugging Face reference outputs with identical argmax results across 334 positions.
Quantization to q8_0 retains 95.8% top-1 accuracy, with remaining errors confined within the HF top-5.

This addition allows users to run HRM-text models on local hardware, though it comes with significant performance trade-offs due to the architecture's design.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 2h ago · 1 view

llama.cpp b11001 fixes int16 transpose corruption on RISC-V

The llama.cpp project released build b11001, which includes a critical fix for the `permute_transpose_impl` function handling int16 data on RISC-V architectures.

github llama.cpp · 8h ago · 1 view

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

github llama.cpp · 16h ago · 2 views

llama.cpp b10989 enables AllReduce for ROCm and provides multi-platform binaries

The llama.cpp project has released version b10989, which introduces support for enabling AllReduce operations within the ROCm backend. This update is part of the ongoing development to expand hardware compatibility and distributed computing capabilities for large language model inference.

github llama.cpp · 21h ago · 1 view

llama.cpp b10988 adds OpenCL MoE expert matmul selection for speculative decoding

The llama.cpp project released build b10988, introducing changes to the OpenCL backend specifically for Mixture of Experts (MoE) models. The update implements logic to choose the MoE expert matrix multiplication based on batch size during speculative decoding and multi-token prediction.

github llama.cpp · 1d ago · 1 view

llama.cpp b10984 adds CUDA row-contiguous SUM_ROWS support

The llama.cpp project released version b10984, introducing CUDA support for row-contiguous SUM_ROWS operations. This update also organizes the code to include GGML_OP_MEAN for handling row-contiguous tensors via a shared kernel.