github llama.cpp · 2d ago · inference

llama.cpp b10684 improves SYCL --fit VRAM accounting

from English

The llama.cpp project released version b10684, which includes a specific fix for the SYCL backend to make the `--fit` flag respect `--fit-target` more accurately.

The `--fit` algorithm now accounts for actual peak VRAM required for a given context size on SYCL backends.
This prevents out-of-memory errors by properly calculating VRAM usage when the allocated context is fully used.
It also stops overly-conservative calculations that previously reserved too much VRAM.
Testing on an Arc b70 with unsloth's qwen3.8 (Q4_K_XL) demonstrated a usable 262144 context with q8_0 KV and MTP using `--fit-target 1`.

This change allows users to utilize larger contexts on SYCL hardware without running out of memory, while avoiding the waste of reserved VRAM from previous conservative estimates.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · just now · 1 view Live

llama.cpp b10721 fixes WebGPU crash and adds Windows CUDA 13.4 support

The llama.cpp project has released build b10721, which includes a critical fix for the WebGPU backend and expands hardware support on Windows.

github llama.cpp · 2h ago

llama.cpp b10718 extends MOE fusion to specdec and multi-token

The llama.cpp project released version b10718, which includes a significant update to its CUDA backend. The most notable change is the extension of Mixture of Experts (MOE) fusion to support speculative decoding (specdec), addressing previous limitations where MOE glu fusion and topk-router fusion were restricted to processing only one token at a time.

github llama.cpp · 5h ago · 1 view

llama.cpp b10721 fixes WebGPU crash and adds Windows CUDA 13.4 support

llama.cpp b10718 extends MOE fusion to specdec and multi-token

llama.cpp b10715 fuses DFlash encoder into KV cache injection

llama.cpp b10714 tunes Vulkan mat-vec rows for Strix Halo

llama.cpp b10706 adds SWIGLU_CLAMP and Vulkan shader support