github llama.cpp · just now · inference

llama.cpp b10675 adds Vulkan hoisting support for row IDs and expert count in shaders

from English

The llama.cpp project released version b10675, which includes a key update to its Vulkan backend. This release adds hoisting support for row IDs and expert counts within the shader code.

The implementation uses hoisted row IDs in coopmat2.
It addresses review feedback regarding count_experts by using vk_op_count_experts_push_constants instead of raw uint vectors.
The fastdiv trick is applied to the ne00 div/mod operation in count_experts.
Per-expert offsets are computed with subgroupExclusiveAdd where supported, keeping a serial path as a fallback.

This update provides optimized Vulkan builds for macOS, Linux, Windows, and Android across various hardware backends including CUDA, ROCm, and OpenCL.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp adds fa-vec tunings for Apple M4 Pro

The llama.cpp project has added fast attention vector (fa-vec) tunings for the Apple M4 Pro GPU via a new Metal backend contribution. This update follows previous work to support additional Apple GPUs and was generated using the `ggml-metal-tuning` tool on a 14-inch MacBook Pro from November 2024.

github llama.cpp · 2d ago · 4 views

llama.cpp b10643 adds multi-NPU support and fully async Hexagon backend

The llama.cpp project released version b10643, introducing support for multi-NPU devices (IQ9, IQ10) and a fully asynchronous backend for the Hexagon architecture.

media Hugging Face Forums · 2d ago · 4 views

EdgeFirst publishes Rust pipeline validating YOLO models on 20 edge platforms

EdgeFirst has published the EdgeFirst Model Zoo on Hugging Face, featuring YOLOv5, YOLOv8, YOLO11, and YOLO26 models validated across 20 distinct edge platforms. The validation relies on a single Rust pipeline that handles zero-copy DMA pre-processing, model decoding, and post-processing directly on the target hardware.

github llama.cpp · 3d ago · 2 views

llama.cpp b10628 adds Apple RDMA RPC transport support

The llama.cpp project has released version b10628, introducing support for Apple RDMA as a remote procedure call (RPC) transport mechanism.

lab OpenAI News · 3d ago · 13 views

OpenAI reports Jalapeño chip delivers up to 3.6x lower latency and 1.9x higher efficiency

OpenAI has released initial performance results for Jalapeño, its first custom inference chip, demonstrating significant gains in speed and power efficiency compared to existing commercial systems. The company states that the new architecture allows for higher throughput and lower latency simultaneously, addressing a common trade-off in current hardware.