github llama.cpp · 1h ago · inference

llama.cpp v0.5.0 adds HRM-Text support, CUDA conv2d acceleration, and multi-address binding

from English

The llama.cpp project has released version 0.5.0, focusing on backend performance, broader model coverage, and more robust server operation. This update introduces support for new architectures like HRM-Text (DFM Mimir 1B) and MiMo-V2.6, while significantly enhancing computational efficiency through CUDA and Metal optimizations.

Accelerates CUDA conv2d operations using implicit GEMM.
Adds Metal MoE and SSM_CONV fusion optimizations.
Allows the server to bind to multiple addresses via comma-separated TCP addresses and UNIX sockets.
Enables image outputs from function calls by adding input_image support.
Updates ggml to v0.25.0, expanding hyper-connection and flash-attention support across backends.

The release improves stability for router-spawned children and fixes several chat parser issues, ensuring more reliable operation for users deploying llama.cpp in production environments.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency Local inference

Read original

github llama.cpp · 14h ago · 12 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 1d ago · 11 views

llama.cpp adds DFlash support for HunyuanOCR and fixes draft conversion

The llama.cpp project has added support for DFlash speculative decoding with the HunyuanOCR model by exposing layer input tensors in the target graph. This change allows the draft model to read residual streams, enabling image requests to run successfully with a draft acceptance rate of approximately 0.5 and byte-identical OCR output compared to non-speculative runs.

lab Hugging Face Blog · 1d ago · 14 views

Transformers adds GGUF support via ggml kernels for local inference

Hugging Face's Transformers library now supports loading GGUF quantized models, leveraging the underlying ggml kernels to bring performance close to llama.cpp. This integration allows developers to run quantized checkpoints directly within the standard PyTorch-based API on supported hardware.

github llama.cpp · 5d ago · 22 views

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

github llama.cpp · 6d ago · 18 views

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.