github llama.cpp · just now · inference

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

from English

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

Extends MMVQ GLU fusion to handle mixed quant types in the SYCL backend.
Adds rms_norm+scale and ssm_conv+silu fusions for SYCL.
Fixes spacing issues and converts a macro to a template function.
Provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU, and NPU backends including CUDA, Vulkan, ROCm, and OpenVINO.

This update enhances performance and compatibility for SYCL-based hardware acceleration within the llama.cpp ecosystem.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 17h ago · 9 views

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.

lab Hugging Face Blog · 21h ago · 10 views

Transformers adds GGUF support via ggml kernels for local inference

Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.

github llama.cpp · 4d ago · 21 views

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

github llama.cpp · 6d ago · 15 views

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.

github llama.cpp · 7d ago · 20 views

llama.cpp b11006 adds Hexagon support for K-Quants Q4_K and Q6_K

The llama.cpp project released version b11006, introducing support for Qualcomm Hexagon processors to handle K-Quantization formats. This update implements specific kernels for Q4_K and Q6_K quantization levels on the Hexagon architecture.