github llama.cpp · 1h ago · inference

llama.cpp b10622 fixes Metal OOM crash on iOS/macOS

from English

The llama.cpp project released build b10622, which includes a critical fix for the Metal backend to prevent out-of-memory crashes. The update addresses an issue where `ggml_metal_buffer_init` returning NULL due to allocation failure was not checked, leading to a hard crash via `EXC_BAD_ACCESS` when dereferencing the null pointer.

metal: Added a null-check for `ggml_metal_buffer_init` results in `ggml_backend_metal_buffer_type_alloc_buffer` to handle OOM conditions gracefully.
cont: Fixed logging issues within the codebase.

This change ensures that memory-constrained devices like iOS can surface diagnosable errors instead of crashing when a model or context exceeds the available Metal budget.

Importance 1/3 Trust 1/3 llama.cpp Inference efficiency

Read original

media r/LocalLLaMA · just now · 1 view Live

Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original

The article introduces Quantization-Aware Healing, a method for creating a compressed 4-bit model that achieves performance superior to its full-precision original.

lab Hugging Face Blog · 1h ago · 1 view Live

Papers with Code uses Hugging Face services for hybrid paper search

Papers with Code has implemented a hybrid search system combining keyword and semantic retrieval to help users find relevant AI research papers. The architecture leverages Hugging Face Jobs, Storage Buckets, and Inference Endpoints to manage the embedding pipeline and serve low-latency queries.

lab Hugging Face Blog · 1h ago · 5 views Live

Multiverse Computing's QAH makes 4-bit GPT-OSS outperform its full-precision original

Multiverse Computing introduces Quantization-Aware Healing (QAH), a method that distills compressed, quantized large language models directly from their original pre-compression teachers. Applied to a GPT-OSS 120B model compressed to 60B parameters and quantized to MXFP4, the approach produces a model that beats its own full-precision bfloat16 version on 7 of 9 benchmarks.

github llama.cpp · 2h ago · 1 view

llama.cpp 0.3.0 adds dots3-note model and tensor-split for DeepSeek 4

llama.cpp 0.3.0 introduces support for the dots3-note multimodal model with a new DSA-ISWA KV cache type, along with multi-token prediction (MTP) support for GLM-4.5-Air. The release also adds tensor-split capabilities for DeepSeek 4 and fixes multi-sequence rollback issues.

github llama.cpp · 6h ago · 1 view

llama.cpp b10618 fixes grammar parsing for literal hyphens in character classes

The llama.cpp project released build b10618, which includes a fix for the grammar parser to correctly handle literal hyphens within character classes. Previously, the `gbnf_escape_char_class()` function escaped the hyphen as "\-", but the `parse_char()` function rejected this escape sequence, causing generated tool-call grammars to fail during parsing.