github llama.cpp · 9h ago · inference

llama.cpp adds GLM-5.3-Flash support with k-pool and MTP fixes

from English

The llama.cpp project has added initial support for the GLM-5.3-Flash (GLM5-Next) model architecture, including migration to hybrid-index memory and early Multi-Token Prediction (MTP) capabilities.

Initial MTP support was added, though stripped for the initial pull request to allow focused review.
The k-pool layout is now kept across ubatches to prevent staleness after sequence edits and shared teardowns.
Recurrent rollback checkpoints are correctly written for conv state and delta net state.
A bug in build_attn_mha stream stride was fixed for non-contiguous queries, ensuring correct multi-stream prefill for GLM5-Next.
Quantization protection filters were cleaned up to remove duplicate entries.

These changes ensure stable inference and correct state management for GLM5-Next models across various hardware backends including CUDA, ROCm, and CPU.

Importance 1/3 Trust 1/3 llama.cpp Zhipu AI Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 4d ago · 10 views

llama.cpp b11207 adds tiled Q4_0 and Q8_0 GET_ROWS support for Hexagon

The llama.cpp project released build b11207, which introduces support for tiled Q4_0 and Q8_0 GET_ROWS operations on the Hexagon backend. This update also includes fixes to macros, register spills, and the DMA pipeline, alongside improvements to kernel selection logic and re-enabled vectorization.

github llama.cpp · 7d ago · 17 views

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.

github llama.cpp · 7d ago · 22 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 8d ago · 22 views

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.

lab Hugging Face Blog · 8d ago · 24 views

Transformers adds GGUF support via ggml kernels for local inference

Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.