github llama.cpp · just now · inference

llama.cpp b10408 adds Q3_K ESIMD kernel and moves control to runtime

from English

The llama.cpp project released version b10408, which introduces a DMMV Q3_K ESIMD kernel for Intel GPUs. This release also includes new Q4_K and Q6_K ESIMD kernels and refactors the codebase to allow ESIMD control at runtime rather than compile time.

Adds DMMV Q3_K, Q4_K, and Q6_K ESIMD kernels.
Moves control of ESIMD from compile-time configuration to runtime.
Uses ESIMD by default when available and fixes potential errors with this default behavior.
Includes explicit unroll directives for ESIMD kernels and tidies up the kernel code.

The update provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, OpenCL, ROCm, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 3d ago

llama.cpp b10353 fixes silent wrong results in CUDA and Metal ROLL kernels

The llama.cpp project released build b10353, which addresses a critical bug in the ggml_roll operation on CUDA and Metal backends. Previously, permuted (non-contiguous) source tensors produced silently incorrect results because these backends ignored stride information.

github llama.cpp · 6d ago · 1 view

llama.cpp b10321 fixes NORM/RMS_NORM for partial simdgroup row lengths

The llama.cpp project released version b10321, which includes a critical fix for GGML_OP_NORM and GGML_OP_RMS_NORM operations on Metal. The update resolves an issue where threadgroups sized to leave a partial SIMD group resulted in incorrect mean and variance calculations due to dropped partial sums.

github llama.cpp · 7d ago · 6 views

llama.cpp b10293 adds ROCm CI for gfx1151 and fixes RDNA3.5 inference

The llama.cpp project released build b10293, which includes onboarding AMD ROCm continuous integration for the gfx1151 architecture and resolves correctness issues on integrated RDNA3.5 GPUs.

github llama.cpp · 9d ago · 3 views

llama.cpp b10255 extends oneDNN SDPA to non-FP16 KV caches

The llama.cpp b10255 release extends the oneDNN SDPA path to support non-FP16 key-value (KV) caches, including Q4_0–Q8_0 quantized formats and FP32. This update allows the fused systolic kernel to run identically to the native FP16 path by dequantizing or converting K/V tensors to dense FP16 on-device before processing.

github llama.cpp · 10d ago · 1 view

llama.cpp b10246 routes large q6_K lm_head to flat GEMV

The llama.cpp project released version b10246, which includes a fix for OpenCL performance on specific model configurations. The update addresses an issue where the original dimension condition was insufficient for handling large weights.