github llama.cpp · 1h ago · inference

llama.cpp b10798 makes build info output stream configurable

from English

The llama.cpp project released version b10798, introducing a change to make the build information output stream configurable. This update allows `llama_print_build_info` to write to a caller-provided FILE* instead of hardcoding stderr.

The parameter defaults to stderr so existing callers keep their current behavior.
The version command in llama-app now passes stdout, ensuring plain version output goes to the expected location.

This change improves flexibility for developers who need to redirect build information output programmatically.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

github llama.cpp · 10h ago · 1 view

llama.cpp b10796 adds n_expert_used_max to fix Nemotron-3-Puzzle loading

The llama.cpp project released build b10796, which includes a change to the model loader to support models with varying numbers of experts per layer.

arxiv arXiv cs.LG · 11h ago · 1 view

FlashAttention-4 introduces Direct-P for hardware-aware FP4 inference and training

FlashAttention-4 addresses the performance limitations of Blackwell's 4-bit floating-point (FP4) tensor cores by introducing new paths for noncausal inference and causal training. The method, called Direct-P, maps scores directly to FP4 probabilities to bypass softmax conversion overheads that typically dominate once matrix products shrink.

github llama.cpp · 11h ago · 1 view

llama.cpp b10795 fuses SYCL operations

The llama.cpp project released version b10795, which includes a new optimization for the SYCL backend. This update enables the fusion of RMS_NORM+MUL+ADD and ADD+ADD residual chains under the GGML_SYCL_ENABLE_FUSION flag.

github llama.cpp · 17h ago · 3 views

llama.cpp b10791 adds OpenCL quant lm_head and decode GEMV optimizations

The llama.cpp project released version b10791, introducing significant OpenCL optimizations for quantized language model heads and decoding GEMV operations. This update also includes medium-batch GEMM improvements relevant to speculative decoding and multi-token prediction.

github llama.cpp · 18h ago · 1 view

llama.cpp b10792 fixes GBNF grammar generation for empty object schemas

The llama.cpp project has released build b10792, which includes a fix for GBNF grammar generation specifically addressing issues with empty object schemas.