The Metal backend in llama.cpp has been extended to support f16 and bf16 tensor types for the concat operator, in addition to existing f32 and i32 support. This update includes specialized kernel templates, updated pipeline getters, and improved type-based kernel dispatch, with assistance from pi:llama.cpp/Qwen3.6-27B.
llama.cpp releases version b9688, adding model management and SSE realtime updates APIs. The release includes prebuilt binaries for macOS, Linux, Android, Windows, and openEuler, supporting various architectures and acceleration frameworks like Vulkan, CUDA, OpenVINO, and SYCL.
LLaMA.cpp version b9685 introduces SYCL-based dev2dev memcpy functionality, moving GGML_SYCL_DEV2DEV_MEMCPY to runtime table and improving peer-to-peer communication detection. The release includes precompiled binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and APIs including Vulkan, ROCm, OpenVINO, and SYCL (FP32/FP16).
LLaMA.cpp release b9684 introduces a new 3D convolution operation (conv_3d) and includes optimized implementations. The release provides prebuilt binaries for macOS, Linux, Android, Windows, and openEuler across various architectures and hardware acceleration options, including SYCL, Vulkan, CUDA, and OpenVINO.
llama.cpp version b9682 introduces Vulkan support for Linux and Windows, enabling GPU acceleration. The release includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures, with CPU and GPU options including CUDA, OpenVINO, SYCL, and ROCm.
llama.cpp version b9675 enables FP16 support for operations like SQR, SQRT, LOG, SIN, COS, and CLAMP. The release includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures, with support for Vulkan, ROCm, OpenVINO, SYCL (FP16 and FP32), and CUDA 12.4 and 13.3.
llama.cpp releases version b9680 with updated Vulkan support and new binaries for macOS, Linux, Android, Windows, and openEuler. The release includes CPU and GPU variants for multiple architectures, with support for Vulkan, CUDA, OpenVINO, SYCL, and ROCm.
llama.cpp version b9669 adds backend sampling support for Eagle3. The release includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and hardware acceleration options, including Vulkan, CUDA, ROCm, OpenVINO, and SYCL.
llama.cpp release b9670 includes fixes for NVFP4 edge cases in llama-graph, such as moving post-GEMM MUL operations and restricting build_ffn to supported combinations. The release provides binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and backend options, including CUDA, Vulkan, SYCL, and OpenVINO.
llama.cpp release b9667 introduces Vulkan support with S_v=16 via gated_delta_net. It includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures, with options for Vulkan, CUDA 12.4 and 13.3, ROCm, OpenVINO, and SYCL.
llama.cpp version b9665 introduces a new --offline flag for benchmarking. The release includes binary builds for macOS, Linux, Android, Windows, and openEuler across multiple architectures and hardware acceleration options, including Vulkan, CUDA, ROCm, OpenVINO, and SYCL.
LLaMA.cpp release b9663 adds support for OP EXPM1 and all unit test cases for FLOOR, TRUNC, and ROUND. It includes updated binaries for macOS, Linux, Android, Windows, and openEuler, with support for SYCL (FP32 and FP16), Vulkan, CUDA 12.4 and 13.3, and ROCm 7.2, along with an updated UI.
The sycl update extends support for reordered expert tensor handling in MoE MUL_MAT_ID to Q4_K, Q5_K, and Q6_K. Unsupported 3D reorder cases now fallback instead of aborting.
The llama.cpp release b9661 adds GGML_OP_COL2IM_1D support for Vulkan, using a bounded gather loop instead of full-K scan with modulo. It returns nullptr for unsupported types and includes builds for macOS, Linux, Android, Windows, and openEuler across CPU, Vulkan, CUDA, and SYCL.
TokenPilot reduces inference costs by 61% to 87% in both isolated and continuous modes, outperforming prior systems in cost efficiency while maintaining competitive performance. It uses ingestion-aware compaction and lifecycle-aware eviction to preserve prompt cache continuity and minimize token footprints.
KVEraser enables efficient localized context erasing in large language models by replacing only the KV cache states of an erased span with learned steering states. It achieves near-full-recomputation performance on in-domain tasks across 1K to 32K context lengths, with only a 24% latency increase, and outperforms other approximate methods in long-document QA with 3--4x speedup over full recomputation.
A user shares a Docker configuration for running GLM-5.2-FP8 on HGX-H200 hardware using SGLang. The setup achieves 262k context length and 70 tokens per second with 8 tensor parallelism, using a memory fraction of 0.83. The user notes that vLLM official recipes do not work on H200 due to KV cache FP8 quantization limitations on the DSV3 architecture.
Gemma 4 E2B achieves 255 tokens per second in-browser on an M4 Max using WebGPU kernels. The demo and kernels are now available on Hugging Face for public use.
TRELLIS.2 has been ported to run natively on MLX for Apple Silicon. The model supports 512x512 and 1024x1024 image inputs, with generation times of approximately 70 seconds for 512x517 and 300 to 700 seconds for 1024x1024 on an M4 Max with 128GB unified memory.
GLM-5.2, a 744B parameter model from Z.ai, has been evaluated as the top frontend coding model globally, outperforming all Opus versions including Opus 4.8. This achievement is highlighted in third-party evaluations that validate official offline tests, marking a significant milestone for a model of its size, particularly in the competitive frontend coding domain.