github llama.cpp · just now · inference

llama.cpp adds fa-vec tunings for Apple M4 Pro

from English

The llama.cpp project has added fast attention vector (fa-vec) tunings for the Apple M4 Pro GPU via a new Metal backend contribution. This update follows previous work to support additional Apple GPUs and was generated using the `ggml-metal-tuning` tool on a 14-inch MacBook Pro from November 2024.

The tuning process ran successfully in 1 hour, 13 minutes, and 1 second with no other system load.
The contribution supports f16 and q8_0 data types for the M4 Pro architecture.
This release includes binaries for macOS (Apple Silicon and Intel), iOS, Linux (CPU, Vulkan, ROCm, OpenVINO, SYCL), Android, Windows (CPU, CUDA, Vulkan, OpenCL, ROCm, OpenVINO, SYCL), and openEuler.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

github llama.cpp · 1d ago · 3 views

llama.cpp b10643 adds multi-NPU support and fully async Hexagon backend

The llama.cpp project released version b10643, introducing support for multi-NPU devices (IQ9, IQ10) and a fully asynchronous backend for the Hexagon architecture.

media Hugging Face Forums · 2d ago · 4 views

EdgeFirst publishes Rust pipeline validating YOLO models on 20 edge platforms

EdgeFirst has published the EdgeFirst Model Zoo on Hugging Face, featuring YOLOv5, YOLOv8, YOLO11, and YOLO26 models validated across 20 distinct edge platforms. The validation relies on a single Rust pipeline that handles zero-copy DMA pre-processing, model decoding, and post-processing directly on the target hardware.

github llama.cpp · 3d ago · 2 views

llama.cpp b10628 adds Apple RDMA RPC transport support

The llama.cpp project has released version b10628, introducing support for Apple RDMA as a remote procedure call (RPC) transport mechanism.

lab OpenAI News · 3d ago · 13 views

OpenAI reports Jalapeño chip delivers up to 3.6x lower latency and 1.9x higher efficiency

OpenAI has released initial performance results for Jalapeño, its first custom inference chip, demonstrating significant gains in speed and power efficiency compared to existing commercial systems. The company states that the new architecture allows for higher throughput and lower latency simultaneously, addressing a common trade-off in current hardware.

github llama.cpp · 7d ago · 16 views

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

The llama.cpp project released build b10534, which introduces hardware-specific switch points to tune the crossover between vector and tensor-core paths during quantized decoding on CUDA.