github llama.cpp · 2h ago · inference

llama.cpp b10628 adds Apple RDMA RPC transport support

from English

The llama.cpp project has released version b10628, introducing support for Apple RDMA as a remote procedure call (RPC) transport mechanism.

Adds Apple RDMA as an RPC transport option.
Removes the set_tensor micro optimization and RPC socket pinning per CR.
Removes transparent reconnect functionality.
Configures CI to trigger Apple builds on RPC changes.
Provides binaries for macOS, iOS, Linux, Windows, Android, and openEuler across CPU, GPU, and specialized hardware backends.

Importance 1/3 Trust 1/3 llama.cpp Hardware & chips Inference efficiency

Read original

lab OpenAI News · 6h ago · 2 views

OpenAI reports Jalapeño chip delivers up to 3.6x lower latency and 1.9x higher efficiency

OpenAI has released initial performance results for Jalapeño, its first custom inference chip, demonstrating significant gains in speed and power efficiency compared to existing commercial systems. The company states that the new architecture allows for higher throughput and lower latency simultaneously, addressing a common trade-off in current hardware.

github llama.cpp · 5d ago · 13 views

llama.cpp b10534 adds per-HW CUDA switch points for faster decode

The llama.cpp project released build b10534, which introduces hardware-specific switch points to tune the crossover between vector and tensor-core paths during quantized decoding on CUDA.

lab Hugging Face Blog · 8d ago · 5 views

Constraint-aware GPU allocator improves utilization and priority-weighted output over FIFO

A constraint-aware GPU allocator was benchmarked against a FIFO scheduler across seven scenarios, achieving up to 33 percentage points higher GPU utilization and up to 105% more priority-weighted output on identical hardware. The system treats real-time inference demand as a fluctuating curve rather than a static reservation, allowing batch-like jobs to fill capacity during troughs while respecting strict constraints like contiguous GPU blocks and non-preemption.

media r/LocalLLaMA · 9d ago · 19 views

NVIDIA serves Qwen 3.8 2.4T on GB300 NVL72 at 4K tokens/s per GPU

NVIDIA has demonstrated serving the Qwen 3.8 2.4T parameter model with configurable reasoning capabilities on its GB300 NVL72 hardware platform.

github llama.cpp · 18d ago · 15 views

llama.cpp b10306 adds SYCL GLU flat path and consolidates kernels

The llama.cpp b10306 release introduces performance optimizations for the SYCL backend, specifically targeting Gated Linear Unit (GLU) operations. The update adds a contiguous fast path for fused GLU ops and consolidates previously distinct kernels to share a common launcher.