The llama.cpp project has released version b9909, which introduces timing and progress information to the /responses API stream. This update is available across multiple platforms including macOS, Linux, Windows, Android, and openEuler.

  • The primary functional change is the addition of timings and progress details in the streaming output for the /responses API endpoint.
  • Pre-built binaries are provided for various architectures such as Apple Silicon, Intel x64, ARM64, and s390x.
  • GPU support includes CUDA 12.4 and 13.3, Vulkan, ROCm 7.2, OpenVINO, SYCL, and HIP.
  • KleidiAI support for macOS Apple Silicon is disabled in this release.

This update allows users to monitor the progress of API responses more effectively.