The llama.cpp project released version b10032, which introduces a new CUDA implementation for the GGML_OP_LIGHTNING_INDEXER operation. This update includes both generic vector and WMMA kernels to handle indexing tasks on NVIDIA hardware.

  • The CUDA lightning indexer uses generic vector and WMMA kernels.
  • Template parameters WARPS_PER_BLOCK and K_VECS_PER_BLOCK were added to reduce constant duplication.
  • MMA architecture requirements were relaxed to support Turing GPUs.
  • Alignment checks for Q and K tensors were added to the implementation.
  • The function ggml_cuda_op_lightning_indexer was renamed to ggml_cuda_lightning_indexer.

This release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and other backends.