github llama.cpp · 3h ago · inference

llama.cpp b10997 broadens MoE tile heuristic for RDNA3.5

from English

The llama.cpp project released version b10997, which includes a change to the Mixture of Experts (MoE) ncols_opt tile heuristic to support the RDNA3.5 architecture.

The code in ggml/src/ggml-cuda/mmq.cu was updated to include RDNA3.5 in the condition check.
Performance tests on an AMD Radeon 8060S showed a +16.198% speedup for LFM2.5-8B-A1B-UD-Q4_K_M and +6.189% for Qwen1.5-MoE-A2.7B-Q2_K.
The release provides binaries for macOS, Linux, Windows, Android, and openEuler across CPU, CUDA, ROCm, Vulkan, OpenVINO, and SYCL backends.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

github llama.cpp · 3h ago · 1 view

llama.cpp b10994 fixes Metal NaN in mul_mm_id for large activations

The llama.cpp b10994 release includes a fix for the Metal backend that resolves NaN outputs in the `mul_mm_id` operation when activation values exceed the f16 range. This defect previously caused models like Mistral Small 4 to produce entirely NaN vocabularies during prefill steps of 32 tokens or more on Apple Silicon devices.

media Hugging Face Forums · 2d ago · 5 views

MOLT achieves 23% faster Qwen 1.5B QLoRA fine-tuning than Unsloth on RTX 4060 Laptop

The author has built MOLT, a Windows-first runtime for thermally aware QLoRA fine-tuning on consumer NVIDIA GPUs, and benchmarked it against Unsloth using one million Qwen 1.5B training targets on an RTX 4060 Laptop GPU.

media r/LocalLLaMA · 4d ago · 7 views

Custom code makes DeepSeek V4.1 run faster than official API on A100 GPUs

A user has developed a custom implementation that allows the DeepSeek V4.1 model to run on NVIDIA A100 GPUs with performance exceeding the official API.

media Hugging Face Forums · 4d ago · 3 views

Proposal for vendor-neutral AI runtime with memory-budget-aware execution

A proposal on the Hugging Face forums outlines a common GPU execution and memory-management layer designed to decouple application code from specific hardware vendors like NVIDIA, AMD, and Intel. The goal is to allow users to specify a memory budget rather than manually configuring backend-specific flags, device maps, or offloading strategies.

lab NVIDIA Research · 5d ago · 6 views

Onyx achieves 1.7-9.9x lower cost and 2.3-12.3x lower latency for disk-oblivious ANN search

Researchers propose Onyx, a cost-efficient approach to disk-oblivious approximate nearest neighbor (ANN) search that balances bandwidth and access count by inverting the design of state-of-the-art ORAM-ANN systems.