github vLLM · just now · inference

vLLM v0.28.0rc2 adds DFlash2 Spec Decode with local convolution and candidate selector

from English

The vLLM project released version 0.28.0rc2, which includes the DFlash2 speculative decoding feature.

DFlash2 implements a local convolution mechanism for efficient token generation.
The update introduces a dedicated candidate selector to improve the spec decode process.
This change was cherry-picked from commit b389ac2 and signed off by khluu.

Importance 1/3 Trust 2/3 vLLM Inference efficiency

llama.cpp b10539 fixes Vulkan FA MMQ overflow and adds Windows CUDA 13 support

The llama.cpp project has released version b10539, which addresses a critical numerical stability issue in the Vulkan backend while expanding hardware support for Windows users.

github llama.cpp · 3h ago · 1 view

llama.cpp b10541 adds --mmproj-device argument for multimodal projects

The llama.cpp project released build b10541, introducing the `--mmproj-device` command-line argument to allow users to specify the device backend for loading multimodal projection files.

github llama.cpp · 3h ago · 1 view

llama.cpp b10545 fixes Metal matmul out-of-bounds reads for K not multiple of 32

The llama.cpp project released build b10545, which addresses a critical bug in the Metal backend's tensor API matrix multiplication kernel. The update prevents undefined behavior and corrupted results when processing tensors where the K dimension is not a multiple of 32.

github llama.cpp · 4h ago · 1 view

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

The llama.cpp project released build b10534, which introduces hardware-specific switch points to tune the crossover between vector and tensor-core paths during quantized decoding on CUDA.

github llama.cpp · 5h ago · 1 view

llama.cpp b10538 release with dequantized KV cache for large batches

The llama.cpp project has released version b10538, introducing a key optimization to the metal backend. This update enables dequantization of the key-value (KV) cache exclusively for large batch sizes.