The llama.cpp project has released version b10199, introducing new capabilities to its server component. The primary update enables the server to support embeddings for generating the next token, specifically adding support for embeddings of sampled tokens.

  • Server now supports embedding generation for sampled tokens to facilitate next-token prediction workflows.
  • A fix was applied to the ~server_batch() destructor.
  • macOS Apple Silicon (arm64) builds with KleidiAI are disabled in this release.
  • Pre-built binaries are available for Linux, Windows, Android, and macOS across CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and OpenCL backends.

This update allows users leveraging the llama.cpp server to access token embeddings directly during generation, which is useful for applications requiring vector representations of output tokens.