github llama.cpp · 17h ago · inference

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

from English

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.

The update also fixes DFlash draft conversion against HunYuan targets, resolving two specific failures: an AttributeError caused by unbound vocab handling and config lookups that incorrectly read draft parameters instead of target configurations. These fixes ensure that converting tencent/HunyuanOCR DFlash drafts succeeds for both version 1.5 and v1.0 checkpoints without altering base model conversions.

These changes enable efficient speculative decoding for Hunyuan-based models in llama.cpp, improving inference speed while maintaining output fidelity.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

lab Hugging Face Blog · 20h ago · 10 views

Transformers adds GGUF support via ggml kernels for local inference

Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.

github llama.cpp · 4d ago · 21 views

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

github llama.cpp · 6d ago · 15 views

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.

github llama.cpp · 7d ago · 20 views

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.

github llama.cpp · 8d ago · 26 views

llama.cpp b10981 fixes stateful decode for Gemma-4 and optimizes GPU MoE inference

The llama.cpp b10981 release addresses critical correctness issues in the OpenVINO backend's stateful decoding path, specifically fixing support for models like Gemma-4 that use per-layer head sizes. It also introduces optimizations for GPU Mixture of Experts (MoE) inference and improves performance through KV state relayouts.