github llama.cpp · 1h ago · inference

llama.cpp b10178 adds trace logging for slot similarity checking

from English

The llama.cpp project released build b10178, which introduces trace logging for slot similarity checking within the server component. This update adds detailed logging in `server-context.cpp` during prompt cache slot selection, specifically capturing skip reasons and similarity calculation details.

The release provides binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu with CPU, Vulkan, ROCm, OpenVINO, and SYCL backends), Android, Windows (CPU, CUDA 12/13, Vulkan, OpenCL, OpenVINO, SYCL, HIP), and openEuler. The KleidiAI support for macOS Apple Silicon is disabled in this build.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 5h ago · 1 view

llama.cpp b10174 adds NextN/MTP speculative decoding support for GLM-5.2

The llama.cpp project released build b10174, introducing NextN/MTP speculative decoding support for the GLM_DSA (GLM-5.2) model architecture.

media Together AI Blog · 16h ago

Together AI explains capacity-aware routing for dedicated model inference

Together AI details the architecture of its Dedicated Model Inference platform, which links endpoints, deployments, and immutable configs through a capacity-aware traffic split. This system routes requests based on effective capacity (weight multiplied by ready replicas) rather than fixed percentages, enabling features like zero-downtime rollouts and A/B testing.

media Hugging Face Forums · 17h ago

Attention-Weighted Memory Graphs uses Jina attention for graph-RAG traversal

A new proposal outlines an LLM memory structure that combines late chunking with attention-weighted edges to enable step-by-step traversal. The system leverages Jina AI's embedding model, which calculates attention between sentences, to derive edge weights from the document's internal processing.

github llama.cpp · 17h ago

llama.cpp b10171 fixes Adreno OpenCL multi-stream garbage output

The llama.cpp b10171 release addresses a critical bug in the Adreno KQ/KQV image kernels that caused garbage output during multi-sequence batches. The issue arose because these kernels ignored dimension 3, leading to incorrect reads when using the unified KV cache with multiple streams.

media r/LocalLLaMA · 18h ago

Microsoft releases Mage-VL, a codec-native streaming multimodal model

Microsoft has released Mage-VL, an efficient 4B-parameter multimodal foundation model for image and video understanding that uses a codec-native approach to streamline visual processing. The system separates video streams into anchor (I) frames and predicted (P) frames, retaining only patches where the codec allocates bits to reduce visual token consumption by over 75%.