github llama.cpp · 2h ago · inference

llama.cpp b10107 fixes Windows crash and provides binaries for multiple platforms

from English

The llama.cpp project has released version b10107, which includes a fix for a Windows crash that occurred when the op_poll feature was enabled.

Resolves a crash on Windows when op_poll is active (PR #26029).
Provides pre-built binaries for macOS (Apple Silicon and Intel), Linux (Ubuntu x64, arm64, s390x with CPU, Vulkan, ROCm 7.2, OpenVINO, SYCL), Android (arm64), Windows (CPU, OpenCL Adreno, CUDA 12/13, Vulkan, OpenVINO, SYCL, HIP), and openEuler.
Disables KleidiAI support for macOS Apple Silicon in this release.

This update ensures stability on Windows systems utilizing op_poll while continuing to offer comprehensive hardware support across major operating systems and accelerators.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 7h ago

llama.cpp b10105 refactors mlock/mmap/directio into a single load-mode argument

The llama.cpp project released build b10105, which introduces a significant refactor of the command-line interface by consolidating separate memory management flags into a unified `load-mode` argument.

github llama.cpp · 17h ago

llama.cpp b10099 improves NVFP4 W4A4 quantization on CUDA

The llama.cpp project released version b10099, which includes a significant update to the CUDA backend for improved NVFP4 W4A4 activation quantization.

github llama.cpp · 19h ago

llama.cpp b10098 updates Hexagon activation ops

The llama.cpp project released version b10098, which includes an update to the Hexagon activation operations. This release provides binaries for macOS, Linux, Android, Windows, and openEuler across various CPU and GPU backends.

media Together AI Blog · 21h ago

Together launches Dedicated Model Inference with custom training beta

Together has released a significant update to its inference platform, introducing "Dedicated Model Inference" designed to give users complete control over performance, cost, and quality for open-weight models. The update also announces a closed beta for custom training capabilities, including full-weight and LoRA reinforcement learning.

github llama.cpp · 22h ago · 1 view

llama.cpp b10094 infers speculative type from draft repo sidecars

The llama.cpp project released version b10094, introducing a feature that automatically detects the speculative decoding type from draft repository sidecars. Previously, when using the `-hfd` flag to point to a repository containing mtp-, dflash-, or eagle3- sidecars without explicitly providing `--spec-type`, the system would incorrectly resolve the draft as a full model.