github llama.cpp · just now · inference

llama.cpp b10164 adds chunked SSD matmul for Mamba-2 prefill acceleration

from English

The llama.cpp project released version b10164, introducing a new chunked SSD matrix multiplication implementation designed to accelerate the prefill phase of Mamba-2 models.

ggml-cuda: add chunked SSD matmul for Mamba-2 prefill acceleration
CUDA SSD fixes correctness by promoting s0_stride_seq to int64_t and improves memory coalescing in ssm_ssd_prepare_dt_kernel
Fused M matrix materialization into the pre_matmul kernel
Fixed sdata read-write race in the prepare_dt fallback scan loop
Added SSD CICD fixes for CUDA, HIP, MUSA, and MSVC

This update provides optimized GPU kernels for specific state-space model architectures while maintaining broad compatibility across CPU, Vulkan, ROCm, OpenVINO, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

github llama.cpp · 4h ago · 1 view

llama.cpp b10159 adds FWHT kernel for Metal backend

The llama.cpp project released build b10159, which introduces a Fast Walsh-Hadamard Transform (FWHT) kernel for the Metal backend. This update includes work-in-progress implementation details along with shape guards and code formatting adjustments.

github llama.cpp · 6h ago · 2 views

llama.cpp b10158 adds eagle3-v3 support for gpt-oss model

The llama.cpp project has released build b10158, which introduces support for the eagle3-v3 architecture within the gpt-oss model.

github llama.cpp · 5d ago · 2 views

llama.cpp b10098 updates Hexagon activation ops

The llama.cpp project released version b10098, which includes an update to the Hexagon activation operations. This release provides binaries for macOS, Linux, Android, Windows, and openEuler across various CPU and GPU backends.

lab Hugging Face Blog · 5d ago · 1 view

Diffusers adds native Nunchaku Lite 4-bit inference support

Hugging Face Diffusers now supports loading Nunchaku Lite checkpoints natively via `from_pretrained()`, eliminating the need for a separate inference engine or local CUDA compilation. This integration utilizes the SVDQuant method to run transformer layers with 4-bit weights and activations (W4A4), significantly reducing memory usage while improving inference speed.

github llama.cpp · 6d ago · 1 view

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.