github llama.cpp · 1h ago · inference

llama.cpp b10098 updates Hexagon activation ops

from English

The llama.cpp project released version b10098, which includes an update to the Hexagon activation operations. This release provides binaries for macOS, Linux, Android, Windows, and openEuler across various CPU and GPU backends.

Optimized all-in-one geglu microkernel (hex-geglu)
Enabled non-contiguous source and strided DMA for hex-geglu
Enabled non-contiguous source and strided DMA for the rest of the ACT ops
Generalized GLU per-thread functions via DEFINE_GLU_PER_THREAD macro
Moved UNARY_SILU and UNARY_GELU to unary-ops
Replaced generic ops_context scratchpad usage with local htp_vtcm_layout computation

Importance 1/3 Trust 1/3 llama.cpp Hugging Face Inference efficiency

Read original

lab Hugging Face Blog · 14h ago · 1 view

Diffusers adds native Nunchaku Lite 4-bit inference support

Hugging Face Diffusers now supports loading Nunchaku Lite checkpoints natively via `from_pretrained()`, eliminating the need for a separate inference engine or local CUDA compilation. This integration utilizes the SVDQuant method to run transformer layers with 4-bit weights and activations (W4A4), significantly reducing memory usage while improving inference speed.

github llama.cpp · 1d ago · 1 view

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

github llama.cpp · 2d ago

llama.cpp b10085 fixes Qwen3-VL grounding by aligning position embedding interpolation

The llama.cpp project released build b10085, which includes a fix for the Qwen3-VL vision model's position embedding interpolation to resolve grounding coordinate mismatches.

github llama.cpp · 4d ago · 2 views

llama.cpp b10069 adds OpenCL broadcast support for Adreno MUL_MAT

The llama.cpp project released build b10069, which includes specific OpenCL updates for Qualcomm Adreno GPUs. The primary change involves supporting broadcast operations for the Adreno MUL_MAT kernel and honoring view offsets for the Q8_0 MUL_MAT variant to enable multi-stream functionality in llama-server.

github llama.cpp · 5d ago · 1 view

llama.cpp b10068 rotates injected K/V cache for DFlash

The llama.cpp project released version b10068, which introduces a change to the DFlash model implementation. The update modifies src/models/dflash.cpp to rotate the injected key/value (K/V) cache when K/V quantization is used.