github llama.cpp · just now · inference

llama.cpp b10643 adds multi-NPU support and fully async Hexagon backend

from English

The llama.cpp project released version b10643, introducing support for multi-NPU devices (IQ9, IQ10) and a fully asynchronous backend for the Hexagon architecture.

The Hexagon backend now uses non-host buffers by default and implements full asynchronicity.
Support for Snapdragon Android platform 34 is added.
Q8_0 quantization gains support for inplace dequantizers, DMA pipelines, and row operations.
ALLREDUCE operations are optimized with fused kernels and improved DMA efficiency.
Sync tokens and fence mechanisms are enhanced to synchronize NPU devices across async splits.

These changes enable efficient inference on multi-NPU Snapdragon hardware by allowing concurrent device execution and faster tensor copies.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

media Hugging Face Forums · 16h ago · 2 views

EdgeFirst publishes Rust pipeline validating YOLO models on 20 edge platforms

EdgeFirst has published the EdgeFirst Model Zoo on Hugging Face, featuring YOLOv5, YOLOv8, YOLO11, and YOLO26 models validated across 20 distinct edge platforms. The validation relies on a single Rust pipeline that handles zero-copy DMA pre-processing, model decoding, and post-processing directly on the target hardware.

github llama.cpp · 1d ago · 1 view

llama.cpp b10628 adds Apple RDMA RPC transport support

The llama.cpp project has released version b10628, introducing support for Apple RDMA as a remote procedure call (RPC) transport mechanism.

lab OpenAI News · 2d ago · 8 views

OpenAI reports Jalapeño chip delivers up to 3.6x lower latency and 1.9x higher efficiency

OpenAI has released initial performance results for Jalapeño, its first custom inference chip, demonstrating significant gains in speed and power efficiency compared to existing commercial systems. The company states that the new architecture allows for higher throughput and lower latency simultaneously, addressing a common trade-off in current hardware.

github llama.cpp · 6d ago · 14 views

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

The llama.cpp project released build b10534, which introduces hardware-specific switch points to tune the crossover between vector and tensor-core paths during quantized decoding on CUDA.

lab Hugging Face Blog · 9d ago · 6 views

Constraint-aware GPU allocator improves utilization and priority-weighted output over FIFO

A constraint-aware GPU allocator was benchmarked against a FIFO scheduler across seven scenarios, achieving up to 33 percentage points higher GPU utilization and up to 105% more priority-weighted output on identical hardware. The system treats real-time inference demand as a fluctuating curve rather than a static reservation, allowing batch-like jobs to fill capacity during troughs while respecting strict constraints like contiguous GPU blocks and non-preemption.