github llama.cpp · 1h ago · inference

llama.cpp b11370 fuses shared experts into MMVQ

from English

The llama.cpp project has released build b11370, which includes a CUDA optimization to fuse shared experts into the Mixture of MoE with Vector Quantization (MMVQ) architecture. This change is part of pull request #29184 and also addresses buffer null checks and argument handling for stride_col_dst.

Fuses shared experts into MMVQ for CUDA backends.
Adds null checks for buffers to improve stability.
Moves stride_col_dst to fusion arguments.

This release provides updated binaries for macOS, Linux, Windows, Android, and iOS across various hardware accelerators including CPU, Vulkan, ROCm, OpenVINO, SYCL, and Snapdragon.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

github llama.cpp · 1d ago · 2 views

llama.cpp b11335 routes sm70 to Turing MMVQ table for Volta performance

The llama.cpp project released version b11335, which includes a CUDA optimization for Volta GPUs. The update routes the sm_70 architecture to the existing Turing MMVQ nwarps table instead of falling through to the generic configuration.

media Hugging Face Forums · 2d ago · 2 views

ThermaCompute AI releases CrashLens for local GPU crash log triage

ThermaCompute AI has released CrashLens, a free, local tool designed to help engineers investigate GPU crashes by analyzing log files. The tool identifies known failure patterns within the logs and suggests specific next steps for investigation.

github llama.cpp · 3d ago · 4 views

llama.cpp b11280 release reduces tensor allreduce sync with pinned host buffers

The llama.cpp project has released version b11280, which includes a change to reduce tensor allreduce synchronization by utilizing pinned host buffers.

media Hugging Face Forums · 3d ago · 3 views

AI Hardware Fit connects model VRAM requirements to GPU selection

The creator of AI Hardware Fit has released a free, open-source browser tool designed to help users determine if a local model fits on their GPU and compare suitable hardware options. The tool addresses the complexity of piecing together model files, quantization, context length, and runtime support from disparate sources.

github llama.cpp · 6d ago · 10 views

llama.cpp b11203 adds F16 input support to CUDA FWHT

The llama.cpp project released version b11203, which includes a change to the CUDA Fast Walsh-Hadamard Transform (FWHT) to accept F16 inputs. Previously, the CUDA FWHT only accepted F32 inputs; this update makes the source type a template parameter so the kernel reads an F16 source directly without requiring a converted copy.