The llama.cpp b10956 release introduces a radix select algorithm for the SYCL backend to handle large k values in the TOP_K operation, which previously fell back to CPU for k > 32. This change allows GPU-resident processing of large k by using histogram-based passes that keep shared local memory footprint independent of k.

  • The new implementation supports k values up to 2048 and beyond, eliminating backend round-trips for operations like qwen4exp's sparse-attention indexer.
  • Performance improvements include a 4.98x speedup at ne=[131072,1] with k=2048 and up to 118x faster execution in specific batch configurations compared to the CPU fallback.
  • End-to-end testing on 3x Arc Pro B60 with Qwen3.8-Flash-Next shows a throughput increase from 5.91 to 6.05 t/s at d=131072, with perplexity remaining unchanged.
  • The release includes binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL backends.

This update enables efficient large-k top_k operations on SYCL devices, preventing performance bottlenecks in models requiring high k values for token selection.