A controlled study evaluates OpenPangu 1B and 7B models on Huawei Ascend 910B1 NPUs using weight-only and weight-activation quantization methods. Results show 8-bit weight-only quantization is lossless for both models, while 4-bit quantization is practical for 7B but harmful for 1B on reasoning, math, and code tasks. Ultra-low precision methods like 2-bit and binary fail, and W4A4 SmoothQuant produces non-finite perplexity, indicating extreme low-bit compression remains challenging.
arxiv
arXiv cs.AI
·
50d ago
·
src: 55d ago
·
research
Empirical Study of OpenPangu Quantization on Ascend NPUs
from English
Related articles
media
Hugging Face Forums
·
37d ago
·
1 view
Contributor adds Gemma 4 support and ONNX export optimizations to Hugging Face
A contributor is preparing a pull request to add upstream support for the Gemma 4 model family to Hugging Face, alongside optimizations for ONNX export tracing. The changes address architecture-specific limitations during ONNX tracing and correct layer calculations for heterogeneous model variants, including Mixture of Experts (MoE) and Dense configurations.
media
r/LocalLLaMA
·
44d ago
·
2 views
Ornith 35B works reasonably well with Qwen3.6 35B DFlash speculative model
A user reports achieving a 30-40% increase in token generation speed by pairing the Ornith-1.0-35B model as a draft model with Qwen3.6-35B-A3B-DFlash using llama-server.
media
r/LocalLLaMA
·
45d ago
·
19 views
Ornith-1.0-35B GGUF update: native MTP speculative-decode graft + full serving/TTFT/long-context numbers (llama.cpp, tp=1)
This article reports on an update to the Ornith-1.0-35B model, featuring a native MTP draft head grafted onto the IQ4_XS body for self-speculative decoding in llama.cpp. The author provides comprehensive performance metrics including throughput, time-to-first-token (TTFT), and long-context capabilities on a single RTX PRO 6000 Blackwell GPU.
media
r/LocalLLaMA
·
53d ago
·
1 view
Napkin math on collective hosting costs for diffusiongemma in 2026
A cost analysis estimates that hosting diffusiongemma at different user token levels results in monthly costs per user ranging from 1.7€ to 122.8€. The study finds agentic AI usage is economically unsustainable for collective hosting, though costs could decrease with new GPUs or ASICs and a shorter GPU depreciation period.
media
r/LocalLLaMA
·
54d ago
·
3 views
Fixing Long-Context Decode Cliff on Radeon R9700 with vLLM 0.22.1
A long-context decode performance cliff on AMD Radeon AI PRO R9700 (RDNA4) was resolved by enabling AITER Unified Attention in vLLM 0.22.1. The fix involves relaxing a CDNA gate to include RDNA4, disabling other attention backends, and using bf16 KV cache, resulting in significant speedups across all context lengths. FP8 KV is ineffective on this hardware, and the model's native 262K context is fully achievable with bf16, offering ~2.9× concurrency without needing FP8.