github llama.cpp · 1h ago · inference

llama.cpp b10328 adds initial server tool isolation via Docker

from English

The llama.cpp project has released build b10328, which introduces initial support for isolating tools within the server component using Docker. This update includes documentation and code adaptations to separate tool I/O sandboxes and rename relevant configuration flags.

Server: adds initial tool isolation support via Docker (PR #26507)
Updates documentation and adapts get_info functionality
Fixes Python type checking issues
Separates tools_io_sandbox from tools_io_docker
Renames sandbox to isolate and x-tool-docker to x-tool-runtime

The release provides binaries for macOS, Linux, Windows, Android, and iOS across various hardware backends including CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and OpenCL.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now Live

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.

github llama.cpp · 11h ago · 1 view

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

llama.cpp b10327 fixes CUDA thread/block count in quantized cpy kernel

llama.cpp b10313 adds LRU scheduler to the server

llama.cpp b10312 release with server fix for busy models

llama.cpp b10307 fixes SYCL NVFP4 parsing