github llama.cpp · 1h ago · inference

llama.cpp b10174 adds NextN/MTP speculative decoding support for GLM-5.2

from English

The llama.cpp project released build b10174, introducing NextN/MTP speculative decoding support for the GLM_DSA (GLM-5.2) model architecture.

Adds GLM-5.2 NextN/MTP as a --spec-type draft-mtp target, enabling dense MLA attention and MTP context KV setup.
Updates the convert script to support --mtp/--no-mtp export for GlmMoeDsaForCausalLM, allowing users to drop or keep the NextN block during export.
Provides pre-built binaries for macOS (Apple Silicon and Intel), Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, ROCm, SYCL), and openEuler platforms.

This update enables faster inference speeds for GLM-5.2 models by leveraging speculative decoding techniques within the llama.cpp framework.

Importance 1/3 Trust 1/3 llama.cpp Zhipu AI Inference efficiency

Read original

arxiv arXiv cs.CL · 1d ago

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention

PIVOT (Proxy Indexing Via One full-prefix Traversal) is a training-free, drop-in replacement for the DeepSeek Sparse Attention (DSA) indexer that reduces computational redundancy by sharing one prefix scan across a group of nearby queries. Instead of scoring every preceding token for each query individually, PIVOT aggregates the group into a single proxy query to obtain a candidate set, from which top-k tokens are selected for each query.

media r/LocalLLaMA · 20d ago · 1 view

PIVOT shares prefix scans across query groups to accelerate token-level sparse attention

GLM-5.2 4-bit on 4× DGX Spark achieves 70.8% on Terminal-Bench 2.1

GLM 5.2 FP8 with FP8 KV achieves 79.8% on Terminal-Bench 2.1

GLM5.2 performance

GLM5.2 on 5x Pro 6000s and a 5090, an expensive journey