github llama.cpp · 1h ago · inference

llama.cpp v0.1.2 adds CUDA MMVQ optimizations and mtmd SHA256 hashing

from English

The llama.cpp project has released version 0.1.2, which includes a sync with ggml (bumped to 0.20.2) and several updates to the server and UI components.

CUDA: MMVQ nwarps=8 for bs=1 for dense models on DGX Spark.
mtmd: use sha256 for input hashing.
vocab: support integer tokenizer scores.
mtmd: skip thumbnail for non-tiled LFM2 images.
server: save processed mtmd chunks as placeholder.
ui: enforce alphabetical enum member ordering and refactor Built-In Tools naming.

This release also addresses build clean-ups for xcframework + cmake and updates CI processes for pre-release generation.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github MLX · 9h ago · 7 views

MLX v0.32.1 adds head dimension 72 support in Metal full attention

The MLX library released version 0.32.1, introducing support for a head dimension of 72 within its Metal full attention implementation.

lab Hugging Face Blog · 16h ago

Constraint-aware GPU allocator improves utilization and priority-weighted output over FIFO

A constraint-aware GPU allocator was benchmarked against a FIFO scheduler across seven scenarios, achieving up to 33 percentage points higher GPU utilization and up to 105% more priority-weighted output on identical hardware. The system treats real-time inference demand as a fluctuating curve rather than a static reservation, allowing batch-like jobs to fill capacity during troughs while respecting strict constraints like contiguous GPU blocks and non-preemption.

github llama.cpp · 18h ago

llama.cpp b10472 skips UMA override for HIP builds

The llama.cpp project released build b10472, which includes a fix to skip the Unified Memory Architecture (UMA) override for HIP builds. This change addresses an issue where AMD Accelerated Processing Units (APUs) reported inaccurate memory information via hipMemGetInfo.

media Hugging Face Forums · 1d ago · 9 views

SpiralCoreAttention prototype selects token blocks for long-sequence QLoRA fine-tuning

A developer is seeking feedback on SpiralCoreAttention, an experimental training-time context-selection prototype for open-weight LLM fine-tuning. Instead of processing complete long sequences during every forward and backward pass, the system selects specific token blocks from the sequence to train on, while evaluating the resulting adapter on held-out full-context sequences.

github llama.cpp · 1d ago · 17 views

llama.cpp b10456 fixes SYCL thread count for 7x throughput gain on Arc 70

The llama.cpp project released version b10456, which includes a critical fix for the SYCL backend in quantized copy kernel launches. The update adjusts thread and block counts to be proportional to the quantization size, effectively reducing under- and over-subscription issues.