github llama.cpp · 1h ago · inference

llama.cpp b10584 fixes draft context sizing to prevent server 500 errors

from English

The llama.cpp project released build b10584, addressing a critical issue where the draft context size did not match the target context, causing server requests to fail with 500 errors.

The server now ensures the draft context follows the target context's size, preventing decoding failures when slots are filled beyond previous limits.
Memory reservation for the draft model is measured at the largest context the target can take, allowing contexts sharing cells with the target to drop the kv_size override.
The fit function now takes an optional second model into account, ensuring its memory is measured whenever the main context changes to keep the reduce path exact.

This update resolves stability issues in server deployments by aligning draft and target context sizes dynamically.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 18h ago · 2 views

llama.cpp b10568 uses ggml_rope_set_offset() for DeepSeek 2

The llama.cpp project released build b10568, which updates the model handling to use the `ggml_rope_set_offset()` function. This change is partially applied to support DeepSeek 2.

media Hugging Face Forums · 1d ago · 1 view

Celestium Engine adds VRAM optimizations and CPU fallback for SDXL

The Celestium Engine has introduced several upgrades to improve SDXL performance on low-VRAM hardware. These changes focus on memory management, stability, and fallback mechanisms to ensure consistent generation even on constrained systems.

github llama.cpp · 2d ago · 6 views

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 2d ago · 2 views

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.

github llama.cpp · 2d ago · 6 views

llama.cpp b10538 release with dequantized KV cache for large batches

The llama.cpp project has released version b10538, introducing a key optimization to the metal backend. This update enables dequantization of the key-value (KV) cache exclusively for large batch sizes.