github llama.cpp · 1h ago · inference

llama.cpp b10412 enables backend sampling for dflash and dspark

from English

The llama.cpp project has released version b10412, which introduces backend sampling support for both the dflash and dspark backends. This update also enables p_min values greater than 0 within the backend sampling logic and adds corresponding safeguards.

Enables backend sampling for dflash and dspark.
Allows p_min > 0 in backend sampling with added guards.
Provides binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, OpenVINO, SYCL), Windows (CPU, CUDA, Vulkan, OpenVINO, SYCL, ROCm), Android, and openEuler.

This release expands the available hardware acceleration options for llama.cpp users.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · just now Live

llama.cpp b10414 adds TQ2_0 support and optimizes mul_mv kernel

The llama.cpp project released build b10414, introducing support for the GGML_TYPE_TQ2_0 ternary quantization type in the Metal backend. This update also includes optimizations to the contiguous tensor (cont) mul_mv kernel.

lab OpenAI News · 2h ago

OpenAI previews Ultrafast mode running GPT-5.6 Sol up to 14X faster

OpenAI is launching a preview of Ultrafast, a new service tier that runs the GPT-5.6 Sol model up to 14 times faster than Standard processing. Powered by Cerebras hardware, this mode generates up to 750 output tokens per second and is initially available via the OpenAI API.

github llama.cpp · 3h ago

llama.cpp b10410 removes separate fp32 type promotion in gemm non-oneDNN path

The llama.cpp project has released version b10410, which includes a code change to the SYCL backend. This update removes the separate fp32 type promotion in the GEMM non-oneDNN path and instead uses automatic fp16 promotion.

github llama.cpp · 4h ago

llama.cpp b10408 adds Q3_K ESIMD kernel and moves control to runtime

The llama.cpp project released version b10408, which introduces a DMMV Q3_K ESIMD kernel for Intel GPUs. This release also includes new Q4_K and Q6_K ESIMD kernels and refactors the codebase to allow ESIMD control at runtime rather than compile time.

media Hugging Face Forums · 9h ago · 1 view

liventruth releases UL-SMF for 384x KV-Cache Compression using FSQ

The developer liventruth has open-sourced the Unified Latent-State Memory Fabric (UL-SMF), a hardware-software co-designed architecture that compresses standard FP32 Key-Value caches into a 16-dimensional discrete latent space using Finite Scalar Quantization (FSQ).