The llama.cpp project released version b9893, introducing general performance optimizations for OpenCL flash attention decoding. This update adds vectorized flash-attention decode kernels for f16, q8_0, and q4_0 KV cache formats and improves non-flash-attention KQ matrix multiplication kernels.
- DK=512 decode operations are offloaded to the CPU because they are bandwidth-bound and faster there than on the GPU as depth increases.
- Multiquery flash attention kernels are compiled in a minimal program to prevent the Adreno compiler from running out of memory at larger dimensions.
- The default split size for f16-KV decode is reduced from 2048 to 512 to better utilize GPU resources on single-query workloads.
- A crash fix addresses an out-of-range read when block_n is unset for DK=512 decode scenarios.
The release includes binaries for macOS, Linux, Android, Windows, and openEuler across CPU, CUDA, Vulkan, ROCm, and other backends.