github llama.cpp · just now · inference

llama.cpp b10752 adds metallib build support for xcframework

from English

The llama.cpp project has released version b10752, which introduces metallib build support for the iOS XCFramework. This update enables developers to integrate the library into iOS applications using the Metal backend.

macOS Apple Silicon (arm64) binaries are available.
macOS Intel (x64) binaries are provided.
Ubuntu builds include CPU, Vulkan, ROCm 7.14, OpenVINO, and SYCL variants.
Windows releases support CPU, Vulkan, OpenCL Adreno, CUDA 12/13, OpenVINO, SYCL, and ROCm 7.14.
Android arm64 (CPU) binaries are included.

The release provides precompiled binaries for a wide range of platforms and hardware accelerators.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10751 fuses MoE weighted expert reduction for CUDA

The llama.cpp project released build b10751, which introduces a fused kernel for the MoE (Mixture of Experts) weighted expert reduction on CUDA. This optimization replaces the previous baseline that ran two physical kernels with a single weighted-reduction kernel to reduce intermediate global-memory traffic.

github llama.cpp · 4h ago · 1 view

llama.cpp b10751 fuses MoE weighted expert reduction for CUDA

llama.cpp b10750 optimizes KV cache n-gram lookup for 4.9% throughput gain

llama.cpp b10742 adds fa-vec tunings for A18 Pro

llama.cpp b10738 adds SYCL host-pinned memory limit

llama.cpp b10739 adds fa-vec tuning for M2 Max