github llama.cpp · 1h ago · inference

llama.cpp b10413 adds auto-detection of draft spec type from GGUF metadata

from English

The llama.cpp project released version b10413, which introduces automatic detection of the speculative decoding specification type directly from draft GGUF model metadata.

Auto-detects spec type by reading `general.architecture` from the draft GGUF header and mapping tensors like `dflash` and `markov_w1.weight` to `draft-dspark` or `draft-dflash`.
Fixes an issue where loading local draft models without explicitly specifying `--spec-type` resulted in speculative decoding remaining inactive (type NONE).
Moves the spec-type detection logic into the speculative module using RAII patterns and adds logging to inform users when auto-detection occurs.
Provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, ROCm), and openEuler.

This change ensures that speculative decoding activates correctly for local draft models without requiring manual configuration of the specification type.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 5h ago · 1 view

llama.cpp b10410 removes separate fp32 type promotion in gemm non-oneDNN path

The llama.cpp project has released version b10410, which includes a code change to the SYCL backend. This update removes the separate fp32 type promotion in the GEMM non-oneDNN path and instead uses automatic fp16 promotion.

github llama.cpp · 7h ago · 1 view

llama.cpp b10408 adds Q3_K ESIMD kernel and moves control to runtime

The llama.cpp project released version b10408, which introduces a DMMV Q3_K ESIMD kernel for Intel GPUs. This release also includes new Q4_K and Q6_K ESIMD kernels and refactors the codebase to allow ESIMD control at runtime rather than compile time.

github llama.cpp · 3d ago

llama.cpp b10353 fixes silent wrong results in CUDA and Metal ROLL kernels

The llama.cpp project released build b10353, which addresses a critical bug in the ggml_roll operation on CUDA and Metal backends. Previously, permuted (non-contiguous) source tensors produced silently incorrect results because these backends ignored stride information.

github llama.cpp · 6d ago · 1 view

llama.cpp b10321 fixes NORM/RMS_NORM for partial simdgroup row lengths

The llama.cpp project released version b10321, which includes a critical fix for GGML_OP_NORM and GGML_OP_RMS_NORM operations on Metal. The update resolves an issue where threadgroups sized to leave a partial SIMD group resulted in incorrect mean and variance calculations due to dropped partial sums.

github llama.cpp · 7d ago · 8 views

llama.cpp b10293 adds ROCm CI for gfx1151 and fixes RDNA3.5 inference

The llama.cpp project released build b10293, which includes onboarding AMD ROCm continuous integration for the gfx1151 architecture and resolves correctness issues on integrated RDNA3.5 GPUs.