github llama.cpp · just now · inference

llama.cpp b10889 release avoids V cache allocation for indexer

from English

The llama.cpp project has released build b10889, which includes a memory optimization to avoid allocating the V cache for the indexer since it is not used.

This update provides pre-built binaries for macOS (Apple Silicon and Intel), iOS, Linux (Ubuntu x64, arm64, s390x with CPU, Vulkan, ROCm 10.0, OpenVINO, and SYCL backends), Android (arm64), Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, ROCm 10.0), and openEuler.

The release also includes the llama.cpp UI.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 1h ago · 1 view Live

llama.cpp b10888 adds Vulkan command-buffer debug labels for GPU profilers

The llama.cpp project has released build b10888, which includes a new feature to add command-buffer debug labels for GPU profilers in the Vulkan backend.

github llama.cpp · 3h ago · 1 view

llama.cpp b10886 adds Q1_0 vector intrinsic support for s390x

The llama.cpp project released build b10886, which introduces Q1_0 vector intrinsic support for the s390x architecture. This update includes the implementation of `ggml_vec_dot_q1_0_q8_0` and updates documentation to reflect the new capability.

github llama.cpp · 16h ago · 1 view

llama.cpp b10883 fixes Vulkan matrix multiplication and restores Ampere optimizations

The llama.cpp project released build b10883, which includes significant updates to the Vulkan backend. The primary change involves using spec constants for matrix-matrix multiplication A-type operations to improve shader compilation and performance.

media r/LocalLLaMA · 19h ago · 1 view

gtrg55 releases INT4 quantized NVIDIA Cosmos3 64B for local image generation

A user has published code and weights for running NVIDIA's 64-billion parameter Cosmos3 model locally using INT4 quantization. The implementation supports both Text-to-Image and Image-to-Video tasks on Apple Silicon via MLX as well as CUDA.

github llama.cpp · 1d ago · 2 views

llama.cpp b10872 fixes Jinja null left operand in template lookup

The llama.cpp project released build b10872, which includes a fix for Jinja template handling. The update addresses an issue where templates defaulting an optional variable to none and testing its membership in a map previously hit an error.