The llama.cpp project released build b10043, introducing support for CUDA Virtual Devices. This update disables the NCCL path when virtual devices are utilized and includes code refactoring along with labeled virtual device descriptions.

  • Support for CUDA Virtual Devices is added to the backend.
  • The NCCL communication path is disabled when virtual devices are used.
  • Virtual devices are now labeled in descriptions, and GPUx2 server CI jobs were added.
  • Binaries are provided for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, OpenCL, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.

This release enables users to run llama.cpp on systems utilizing CUDA Virtual Devices while maintaining compatibility across a wide range of hardware architectures and operating systems.