github llama.cpp · 2h ago · inference

llama.cpp b10728 adds CUDA XOR swizzle flash attention and fixes shared memory race

from English

The llama.cpp b10728 release introduces a CUDA implementation of XOR swizzle for Flash Attention K,V shared memory fp16 tiles. This update also resolves a shared memory race condition in Flash Attention on DGX Spark hardware.

Implements XOR swizzle flash attention with fp16 tiles for CUDA.
Fixes use of 64-bit generic pointer instead of 32-bit shared pointer.
Adds swizzle test cases and gates synchronization for the swizzled path only.
Allows swizzling for non-power-of-2 shapes where nbatch_2%32==0.
Refactors Flash Attention swizzle ldmatrix logic into helper functions.

The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

lab Microsoft Research Blog · 2h ago · 1 view

Microsoft releases GigaPath-Flash and GigaTIME-Flash for efficient pathology foundation models

Microsoft Research has released GigaPath-Flash and GigaTIME-Flash, efficient variants of its pathology foundation models designed to enable population-scale discovery by drastically reducing computational requirements.

github llama.cpp · 3h ago · 3 views

llama.cpp b10727 adds concat support for quantized types

The llama.cpp project has released build b10727, which introduces concatenation support for quantized data types. This update is assisted by DeepSeek-V4-Flash-0731 and includes binaries for macOS, Linux, Windows, Android, and openEuler across various hardware backends including CPU, CUDA, ROCm, and Vulkan.

github llama.cpp · 5h ago · 6 views

llama.cpp b10726 speeds up large batch IQ model prompt processing with AVX2

The llama.cpp project released version b10726, which introduces significant performance optimizations for processing large batch sizes in IQ quantized models. The primary update leverages AVX2 instructions to accelerate grid IQ quants through batched GEMM operations.

github llama.cpp · 6h ago · 2 views

llama.cpp b10724 optimizes kv-cache state restore to reduce copy count from 1.3M to 224

The llama.cpp project released build b10724, which significantly improves the performance of restoring non-contiguous KV cache cells by batching scatter reads per contiguous run.

github llama.cpp · 7h ago · 5 views

llama.cpp b10723 tunes OpenCL quant paths for Intel Xe-LP GPUs

The llama.cpp project released version b10723, which includes specific optimizations for OpenCL on Intel Xe-LP graphics hardware. The update focuses on improving tensor generation (TG) and prefill (PP) performance by adjusting quantization kernels.