github llama.cpp · just now · inference

llama.cpp b11140 enables sparse-fa for dsv4 prefill on CUDA

from English

The llama.cpp project released version b11140, which includes a CUDA optimization enabling sparse-fa for dsv4 prefill. This update addresses performance issues in the flash attention mask scanning process by templating kernels to allow compile-time loop unrolling.

The query loop of `flash_attn_mask_to_sparse_indices` is now bounded at compile time, reducing scan time from 46 to 17 microseconds for sparse decode shapes with 49k columns.
Out-of-bounds checks for the sparse mask scan are moved to host code, allowing the batched sparse operation at 49k context to drop from 586 to 244 microseconds.

This release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, GPU, and NPU backends.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago · 5 views Live

llama.cpp b11138 parses unary +/- before variables

The llama.cpp project released build b11138, which updates the Jinja template parser to handle unary plus and minus operators before variables.

github llama.cpp · 2h ago · 12 views

llama.cpp b11136 adds OpenAI video_url support to llama-server

The llama.cpp project released build b11136, which updates the llama-server to accept the OpenAI standard "video_url" content type for video inputs. Previously, the server only accepted a non-standard "input_video" type and rejected data URIs, causing compatibility issues with OpenAI-conformant clients.

github llama.cpp · 3h ago · 12 views

llama.cpp b11132 adds Gemma 4 DSpark draft backbone support

The llama.cpp project released version b11132, introducing support for the Gemma 4 DSpark draft backbone. This update enables GGUF conversion and runtime capabilities for full-attention and SWA Gemma 4 drafts, including tied output weights and boolean backbone metadata.

github llama.cpp · 8h ago · 12 views

llama.cpp b11122 extends MMVQ GLU fusion and adds new SYCL fusions

The llama.cpp project has released version b11122, which includes significant updates to the SYCL backend. The release extends the MMVQ GLU fusion to support mixed quantization types and introduces new fusions for rms_norm+scale and ssm_conv+silu operations.

github llama.cpp · 10h ago · 12 views

llama.cpp b11120 hides Vulkan internal symbols to fix state destruction

The llama.cpp project released version b11120, which addresses a critical bug in the Vulkan backend by hiding internal symbols. This change prevents duplicate-dlopen state destruction issues that occurred when internal symbols were exported.