github llama.cpp · just now · inference

llama.cpp b11000 fixes remote code execution via cached compute graph use-after-free

from English

The llama.cpp project released version b11000, which addresses a critical security vulnerability in the RPC server. The fix resolves a use-after-free bug that allowed unauthenticated remote clients to achieve remote code execution by manipulating cached compute graphs.

A previous implementation cached compute graphs per device to avoid resending tensor data during GRAPH_RECOMPUTE operations.
These cached nodes held direct pointers to backend buffers; if a buffer was freed via FREE_BUFFER, subsequent re-execution accessed dangling pointers.
Attackers could reshape memory chunks via ALLOC_BUFFER and SET_TENSOR commands to leak libc addresses and hijack the buffer interface vtable.
The patch discards all cached graphs in free_buffer(), causing the client to fall back to GRAPH_COMPUTE on the next call.

This update prevents remote code execution exploits targeting the llama.cpp RPC server without requiring any protocol or API changes.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Safety & alignment

Read original

media r/LocalLLaMA · 90d ago · 5 views

Real-world token cost savings from rtk, headroom, and caveman

A real workload analysis shows headroom, rtk, and caveman reduce token costs by 2.8%, 0.5%, and 0.4% respectively, totaling 3.7% of baseline spending. However, savings are limited by payload diversity, with most traffic being plain text or source code, and the tools only compress structured outputs. Most cost reduction occurs on the cheapest token stream—cache reads—while the tools do not affect prompt caching or output costs, and coverage gaps exist, especially for rtk.

github llama.cpp · 4h ago · 1 view

llama.cpp b10997 broadens MoE tile heuristic for RDNA3.5

The llama.cpp project released version b10997, which includes a change to the Mixture of Experts (MoE) ncols_opt tile heuristic to support the RDNA3.5 architecture.

github llama.cpp · 4h ago · 1 view

llama.cpp b10994 fixes Metal NaN in mul_mm_id for large activations

The llama.cpp b10994 release includes a fix for the Metal backend that resolves NaN outputs in the `mul_mm_id` operation when activation values exceed the f16 range. This defect previously caused models like Mistral Small 4 to produce entirely NaN vocabularies during prefill steps of 32 tokens or more on Apple Silicon devices.

github llama.cpp · 5h ago · 1 view

llama.cpp b10995 makes Vulkan MUL_MAT_ID BN/2 tail unconditional

The llama.cpp project released build b10995, which modifies the Vulkan backend's handling of the MUL_MAT_ID operation. Specifically, the change makes the BN/2 tail unconditional by using BN/2 as the default for BNover2 and as the disabled fallback for BNover4, while removing the enable gate from that branch.

lab NVIDIA Research · 12h ago · 4 views

ShareMMU enables secure address translation sharing among untrusted accelerators with low overhead

Researchers present ShareMMU, an IOMMU design that allows multiple untrusted accelerators to securely reuse pre-translated addresses within a shared virtual address space. This approach mitigates side-channel risks associated with large Shared Translation Lookaside Buffers (TLBs) while maintaining high performance.