github llama.cpp · 4d ago · inference

llama.cpp b11207 adds tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon

from English

The llama.cpp project released build b11207, which introduces support for tiled Q4_0 and Q8_0 GET_ROWS operations on the Hexagon backend. This update also includes fixes to macros, register spills, and the DMA pipeline, alongside improvements to kernel selection logic and re-enabled vectorization.

Added tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon using tiled HVX dequantization.
Fixed macros and register spills in hex-get-rows while cleaning up checks for unsupported operations.
Improved the DMA pipeline and simplified kernel selection logic.
Re-enabled the vectorizer after a regression was identified during a sampler update.

The release provides binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, GPU, NPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 10h ago · 2 views

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

github llama.cpp · 7d ago · 17 views

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.

github llama.cpp · 7d ago · 22 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 8d ago · 22 views

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.

lab Hugging Face Blog · 8d ago · 24 views

Transformers adds GGUF support via ggml kernels for local inference

Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.