github llama.cpp · 1h ago · inference

Aaron Teo adds -lm mlock argument to llama.cpp

from English

Aaron Teo has submitted a change to the llama.cpp repository that introduces a new `-lm mlock` command-line argument. This addition allows users to explicitly lock memory in regions where the code performs `mlock` operations but does not use `mmap`.

The commit also includes documentation updates and fixes for automated review points, while reverting previous changes to the meaning of the `--mlock` flag to ensure it applies only to non-mmap'ed mlock scenarios.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

github llama.cpp · 4d ago

llama.cpp b10093 fixes DeepSeek v4 template and adds drop_reasoning flag

The llama.cpp project has released build b10093, which includes specific updates to support DeepSeek v4 models. The release addresses template handling and introduces new configuration flags for this model family.

github llama.cpp · 5d ago · 1 view

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

github llama.cpp · 6d ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.

media Hugging Face Forums · 9d ago · 3 views

zhongkaifu releases TensorSharp, a native .NET LLM inference engine for GGUF models

zhongkaifu has released TensorSharp, an open-source local LLM inference engine built entirely in C# that supports Windows, macOS, and Linux. The engine handles GGUF models from Unsloth, including Gemma 4 and Qwen 3.6 with multimodal capabilities, while providing APIs compatible with OpenAI and Ollama.

github llama.cpp · just now Live

llama.cpp b10148 fixes draft sidecar resolution and log verbosity

The llama.cpp project released build b10148, addressing issues with speculative decoding configuration and logging. The update ensures that explicit command-line arguments for draft models take precedence over automatic sidecar resolution.