github llama.cpp · 1h ago · inference

llama.cpp b10171 fixes Adreno OpenCL multi-stream garbage output

from English

The llama.cpp b10171 release addresses a critical bug in the Adreno KQ/KQV image kernels that caused garbage output during multi-sequence batches. The issue arose because these kernels ignored dimension 3, leading to incorrect reads when using the unified KV cache with multiple streams.

Routes tensors with ne03/ne13 > 1 to the general path that correctly handles dim 3.
Honors view_offs when creating sub-buffers to prevent silent misreads.
Fixes perplexity scores on Adreno 740 and 840 devices, reducing PPL from ~1940 to ~15.6 for Llama-3.2-1B-Instruct Q4_0.
Includes binaries for macOS, iOS, Linux, Android, Windows, and openEuler across CPU, GPU, and specialized backends.

This fix ensures correct inference results on Adreno devices where Flash Attention is disabled or unavailable, preventing performance degradation in multi-slot server configurations.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media Together AI Blog · just now Live

Together AI explains capacity-aware routing for dedicated model inference

Together AI details the architecture of its Dedicated Model Inference platform, which links endpoints, deployments, and immutable configs through a capacity-aware traffic split. This system routes requests based on effective capacity (weight multiplied by ready replicas) rather than fixed percentages, enabling features like zero-downtime rollouts and A/B testing.

media Hugging Face Forums · 1h ago Live

Attention-Weighted Memory Graphs uses Jina attention for graph-RAG traversal

A new proposal outlines an LLM memory structure that combines late chunking with attention-weighted edges to enable step-by-step traversal. The system leverages Jina AI's embedding model, which calculates attention between sentences, to derive edge weights from the document's internal processing.

media r/LocalLLaMA · 3h ago

Microsoft releases Mage-VL, a codec-native streaming multimodal model

Microsoft has released Mage-VL, an efficient 4B-parameter multimodal foundation model for image and video understanding that uses a codec-native approach to streamline visual processing. The system separates video streams into anchor (I) frames and predicted (P) frames, retaining only patches where the codec allocates bits to reduce visual token consumption by over 75%.

lab Hugging Face Blog · 4h ago

Ai2 releases OlmoEarth Platform for planetary-scale geospatial inference

Ai2 has launched the OlmoEarth Platform, an infrastructure system designed to take Earth observation foundation models from fine-tuning through to large-scale inference. The platform addresses the engineering challenges of processing terabytes of multimodal satellite data across multiple providers and resolutions.

github llama.cpp · 5h ago

llama.cpp b10165 adds iq4_nl and q1_0 support to Vulkan FA

The llama.cpp project has released build b10165, which restores iq4_nl quantization support for the Flash Attention (FA) Vulkan backend. This change addresses issue #23681 by re-enabling the feature after concerns about shared memory usage were deemed negligible.