media r/LocalLLaMA · 8d ago

Real-world token cost savings from rtk, headroom, and caveman

A real workload analysis shows headroom, rtk, and caveman reduce token costs by 2.8%, 0.5%, and 0.4% respectively, totaling 3.7% of baseline spending. However, savings are limited by payload diversity, with most traffic being plain text or source code, and the tools only compress structured outputs. Most cost reduction occurs on the cheapest token stream—cache reads—while the tools do not affect prompt caching or output costs, and coverage gaps exist, especially for rtk.

github llama.cpp · 8d ago

llama.cpp Release b9703: Updates and Binary Downloads

llama.cpp version b9703 includes a rework of the server's preset handling, removing remote HF preset support and deprecated functions. The release provides binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and hardware acceleration options, including Vulkan, CUDA, OpenVINO, and SYCL.

github llama.cpp · 8d ago

llama.cpp release b9704: fixes invalid grammar handling and adds new binaries

llama.cpp version b9704 now returns HTTP 400 for invalid grammar instead of silently dropping constraints. The release includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and hardware accelerators, with support for Vulkan, ROCm, OpenVINO, SYCL, and CUDA.

media r/LocalLLaMA · 8d ago

unsloth GLM-5.2-GGUF with 2bit quantization at 238GB

The unsloth GLM-5.2-GGUF model is available with 2bit quantization, sized at 238GB. It is hosted on Hugging Face and shared via a Reddit post in the LocalLLaMA community.

media r/LocalLLaMA · 8d ago

NVFP4 kv cache quantization on sm120 will make 32GB VRAM systems very capable

Qwen3.6-27B runs at ~60 tokens/sec on 32GB VRAM with FP8 KV quantization. NVFP4 kv cache quantization on SM120 could significantly enhance performance on such systems, though current implementation is not yet available.

media r/LocalLLaMA · 8d ago

Llama Bench vs Real-World Performance Discrepancy

The user reports a significant gap between Llama benchmark results and actual model performance. Benchmarks show 754 tk/s prefill and 36 tk/s generation, but real usage reveals only 7.98 tokens per second, with high latency and poor throughput. The discrepancy is attributed to real-world usage conditions, not benchmark settings, suggesting the model's actual performance is far below the benchmarked speed.

github llama.cpp · 8d ago

LLaMA.cpp Release b9698 Adds Self-Update Support and Multiple Platform Binaries

LLaMA.cpp version b9698 enables self-updates only when built with llama-install.sh. The release includes binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures and hardware acceleration options, including Vulkan, CUDA, OpenVINO, and SYCL.

arxiv arXiv cs.LG · 8d ago

TransitNet Achieves 95.2% Accuracy in Low-SNR Transit Searches

TransitNet, a compact attention-augmented deep learning framework, achieves 95.2% accuracy in low-SNR transit blind searches, outperforming TLS and BLS in ROC-AUC and PR-AP values. It recovers 93.0% of injected Earth- and sub-Earth-size transits, with 97.4% of injected transits fully covered by estimated transit windows, and successfully recovers all 34 confirmed Kepler planets with a mean midpoint error of 1.24 hours.

arxiv arXiv cs.LG · 8d ago

EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts

EfficientRollout introduces a self-speculative decoding framework that reduces rollout and end-to-end latency by up to 19.6% and 12.7% respectively, without compromising final model quality. It uses a quantized drafter derived from the target model and integrates a system-aware toggle policy to avoid compute-bound regimes, enabling effective speculation during evolving policy generations.

arxiv arXiv cs.LG · 8d ago

FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers

FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.

arxiv arXiv cs.LG · 8d ago

CAHP: Complementary Attention Head Pruning for Efficient Transformers

CAHP introduces a post-hoc framework that uses graph-theoretical clustering and information-theoretic measures to select complementary attention heads in Transformers. It automatically determines head retention without predefined sparsity, identifying a performance degradation threshold to ensure minimal model loss, and outperforms baselines in high-compression scenarios by preserving functionally critical heads in intermediate layers.

arxiv arXiv cs.AI · 8d ago

SwitchBraidNet: Lightweight EEG Model for Hybrid BCIs

SwitchBraidNet is a quantisation-aware EEG classification architecture that achieves high accuracy in motor imagery and SSVEP tasks. It outperforms four baselines in FP16 and FP32, with MI accuracy of 69.49%, SSVEP accuracy of 93.48%, and a hybrid information transfer rate of 64.82 bits/min in FP16. The model runs efficiently with only 3.03 KB of INT8 storage, enabling low-power embedded deployment.

arxiv arXiv cs.AI · 8d ago

TransitNet Achieves 95.2% Accuracy in Low-SNR Transit Searches

TransitNet, a compact attention-augmented deep learning framework, achieves 95.2% accuracy in low-SNR transit blind searches, outperforming TLS and BLS in ROC-AUC and PR-AP values. It recovers 93.0% of injected Earth- and sub-Earth-size transits, with 97.4% of injected transits fully covered by estimated transit windows, and successfully recovers all 34 confirmed Kepler planets with a mean midpoint error of 1.24 hours.

arxiv arXiv cs.AI · 8d ago

FoMoE Breaks Full-Replica Barrier with Partitioned Expert Layers

FoMoE introduces a system that partitions expert layers across workers to avoid full model replicas, reducing communication costs by up to 1.42x over efficient baselines and 45.44x over DDP. It achieves up to 1.4x throughput speedups via a skip-token mechanism and demonstrates stable routing, with projected benefits extending to 100B-scale models through system modeling.

lab Claude Code Releases · 8d ago

Claude Code v2.1.181 Release Notes

Claude Code v2.1.181 introduces support for setting config settings via prompt syntax like /config thinking=false, adds sandbox Apple Events support on macOS, and improves streaming, auto-retry, and subagent behavior. It also fixes numerous bugs related to startup, file handling, clipboard, and UI responsiveness across platforms.

github llama.cpp · 8d ago

ggml-cpu: Conditionally enable POWER11 backend based on compiler support

The ggml-cpu project now conditionally enables the POWER11 backend in ggml based on compiler support for -mcpu=power11. This prevents build failures on current GCC/Clang toolchains while maintaining forward compatibility. Updates to CMakeLists.txt support this change, and -mcpu=power10 is used for both P10 and P11 architectures.

github llama.cpp · 8d ago

llama.cpp Release b9692 Adds New Binaries and Fixes

llama.cpp version b9692 introduces new binaries for macOS, Linux, Android, Windows, and openEuler across multiple architectures. The release includes updates to support Vulkan, ROCm, OpenVINO, SYCL, and HIP, with fixes to remove batch dim usage in llava_uhd.

github llama.cpp · 8d ago

Metal backend adds f16 and bf16 support for concat operator

The Metal backend in llama.cpp has been extended to support f16 and bf16 tensor types for the concat operator, in addition to existing f32 and i32 support. This update includes specialized kernel templates, updated pipeline getters, and improved type-based kernel dispatch, with assistance from pi:llama.cpp/Qwen3.6-27B.

github llama.cpp · 8d ago

llama.cpp releases version b9688 with new APIs and cross-platform binaries

llama.cpp releases version b9688, adding model management and SSE realtime updates APIs. The release includes prebuilt binaries for macOS, Linux, Android, Windows, and openEuler, supporting various architectures and acceleration frameworks like Vulkan, CUDA, OpenVINO, and SYCL.

media r/LocalLLaMA · 8d ago

llama.cpp GPU memory optimization tips

llama.cpp users can free up GPU memory by disabling mmproj offload, using reduced KV cache types, and adjusting spec-draft-n-max. Parameters like --ctx-checkpoints and --fit-target have minimal impact, while --parallel helps in multi-user setups but not for single users.