github llama.cpp · 1h ago · inference

llama.cpp b10150 adjusts ggml offloading logic and fixes dsv4 graph

from English

The llama.cpp project has released build b10150, which includes adjustments to the ggml logic for offloading operations to the weight's backend. This update also addresses fixes for the llama dsv4 graph.

ggml: adjust logic for offloading ops to weight's backend
llama: dsv4 graph fixes

This release provides updated binaries for macOS, Linux, Windows, Android, and iOS across various hardware backends including CPU, CUDA, Vulkan, ROCm, and OpenVINO.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency Local inference

Read original

media Hugging Face Forums · 3h ago

TraceML adds Hugging Face Trainer callback to diagnose data-loading bottlenecks

The author has integrated a diagnostic tool into TraceML, an open-source PyTorch observability project, to help users distinguish between GPU compute-bound and input-bound training performance issues. This new feature allows developers to quickly identify whether low GPU utilization is caused by slow data loading or actual model computation limits.

github llama.cpp · 5h ago · 1 view

Aaron Teo adds -lm mlock argument to llama.cpp

Aaron Teo has submitted a change to the llama.cpp repository that introduces a new `-lm mlock` command-line argument. This addition allows users to explicitly lock memory in regions where the code performs `mlock` operations but does not use `mmap`.

github llama.cpp · 4d ago

llama.cpp b10093 fixes DeepSeek v4 template and adds drop_reasoning flag

The llama.cpp project has released build b10093, which includes specific updates to support DeepSeek v4 models. The release addresses template handling and introduces new configuration flags for this model family.

github llama.cpp · 5d ago · 1 view

llama.cpp b10089 adds full k-quant and i-quant support to CUDA GET_ROWS

The llama.cpp b10089 release extends CUDA GET_ROWS operations to support k-quants, i-quants, and mxfp4 quantization formats. This change ensures that all quantized GGML types can now take the direct device path for embedding lookups.

github llama.cpp · 6d ago

llama.cpp b10076 vectorizes get_rows with int4 copy for 27% speedup

The llama.cpp project released build b10076, which introduces a performance optimization to the CUDA backend by vectorizing the `get_rows` operation using int4 copies. This change hoists row-invariant work out of the per-element loop and adds a vectorized path that copies 16 bytes per thread for contiguous same-type data.