github llama.cpp · 12h ago · inference

llama.cpp b11035 adds IQ3_S MMQ matmul kernels for Vulkan

from English

The llama.cpp project has released version b11035, introducing support for IQ3_S MMQ matrix multiplication kernels in its Vulkan backend.

The update includes new IQ3_S MMQ matmul kernels for Vulkan.
Block_a_to_shmem now performs 2-byte loads to optimize memory access.
Alignment checks have been adjusted as IQ3_S utilizes the K tile size.
Binaries are available for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, OpenVINO, SYCL, and OpenCL platforms.

This release provides updated binaries for various operating systems and hardware accelerators, enabling users to leverage the new Vulkan quantization support.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b11046 adds OpenCL support for flash_attn_f32_f16_bin kernel

The llama.cpp project released version b11046, which introduces OpenCL support for the `flash_attn_f32_f16_bin` kernel. This update also includes guarded prefill functionality for Flash Attention on OpenCL devices.

github llama.cpp · 1h ago · 1 view Live

llama.cpp b11044 updates hexagon im2col and releases binaries

The llama.cpp project has released version b11044, which includes an update to the hexagon backend for improved image-to-column (im2col) operations. The release provides pre-built binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU, and other hardware accelerators.

github llama.cpp · 2h ago · 1 view

llama.cpp b11045 adds ROLL op support to Hexagon backend

The llama.cpp project has released version b11045, which introduces support for the ROLL operation within the Hexagon backend.

github llama.cpp · 1d ago · 5 views

llama.cpp b11026 skips gate_up_exps for Qwen35MoE

The llama.cpp project released build b11026, which introduces a change to skip gate_up_exps when the TENSOR_SKIP flag is set. This modification is specifically required for Qwen35MoE models where MTP tensors are fused but not loaded.

github llama.cpp · 2d ago · 4 views

llama.cpp b11019 fixes GGUF data alignment for embedded files and adds LoRA loading

The llama.cpp project released version b11019, addressing a critical bug where GGUF files embedded at non-aligned offsets returned incorrect tensor data. The update aligns the GGUF data section relative to the file start rather than offset 0, ensuring correct loading of embedded models.