github llama.cpp · just now · inference

llama.cpp b10792 fixes GBNF grammar generation for empty object schemas

from English

The llama.cpp project has released build b10792, which includes a fix for GBNF grammar generation specifically addressing issues with empty object schemas.

This release provides binaries and frameworks for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64, arm64, s390x), Android, Windows, and openEuler across various hardware backends including CPU, CUDA, Vulkan, ROCm, OpenVINO, SYCL, and OpenCL.

The update ensures correct grammar generation for empty object schemas in GBNF.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

github llama.cpp · 2h ago · 1 view

llama.cpp b10790 tunes MMVQ to MMQ crossover for SM87

The llama.cpp project has released build b10790, which includes a specific tuning of the MMVQ to MMQ crossover for SM87.

media r/LocalLLaMA · 7h ago · 1 view

IFM releases K2-Horizon-MoVA-36B-A4B, a sparse MoE model with 4B active parameters

IFM has released the final checkpoint for K2-Horizon-MoVA-36B-A4B, a Mixture-of-Experts model featuring Mixture-of-Values attention that stores 36B parameters but runs only 4B per token. The release includes GGUF formats and links to smaller variants in the K2-Horizon family.

github llama.cpp · 9h ago · 2 views

llama.cpp adds sparse flash attention support for Metal prefill

The llama.cpp project has introduced sparse flash attention (FA) support for the Metal backend, specifically targeting the prefill phase of model inference. This update includes a new kernel that compacts finite mask entries into per-row index lists and extends the vector FA kernel with optional sparse index gathering.

github llama.cpp · 10h ago · 2 views

llama.cpp b10782 fixes multi-GPU CUDA graph optimization

The llama.cpp project released build b10782, which addresses a critical issue in the CUDA backend for multi-GPU setups. The update allows concurrent streams per split and enables CUDA graph optimization to run correctly across multiple devices.