github llama.cpp · 1h ago · inference

llama.cpp b10330 fuses rms_norm, mul, and rope on CUDA

from English

The llama.cpp project has released version b10330, introducing a CUDA optimization that fuses the rms_norm, multiplication, and RoPE operations into a single kernel. This change is accompanied by memory range checks for the fused operations and new test cases for broadcast weights.

CUDA: fuse rms_norm + mul + rope (+ view + set_rows)
tests: add broadcast weight case to rms_norm_mul_rope
CUDA: check memory ranges before rms_norm rope fusion
CUDA: check memory ranges in rope set_rows fusion

The release provides binaries for macOS, Linux, Windows, Android, and iOS across various hardware backends including CPU, Vulkan, ROCm, OpenVINO, SYCL, and HIP.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 2h ago

llama.cpp b10328 adds initial server tool isolation via Docker

The llama.cpp project has released build b10328, which introduces initial support for isolating tools within the server component using Docker. This update includes documentation and code adaptations to separate tool I/O sandboxes and rename relevant configuration flags.

github llama.cpp · 12h ago · 1 view

llama.cpp b10328 adds initial server tool isolation via Docker

llama.cpp b10327 fixes CUDA thread/block count in quantized cpy kernel

llama.cpp b10313 adds LRU scheduler to the server

llama.cpp b10312 release with server fix for busy models

llama.cpp b10307 fixes SYCL NVFP4 parsing