media Hugging Face Forums · 1h ago · open_models

Dev-4B combines quick calibrated decisions with risk-based reasoning escalation

from English

The author released Dev-4B, a small local model based on Qwen3-4B-Instruct that answers typed questions about documents with calibrated confidence. It uses a built-in router to decide whether to provide a quick answer or escalate to step-by-step reasoning only when the initial prediction is likely wrong.

Evaluated on 7,100 frozen test questions, Dev-4B achieved 81.8% accuracy in 1.37 seconds per question.
This outperforms always-quick answers (76.3%, 0.16 s/q) and always-reasoning approaches (78.0%, 10.2 s/q).
The model uses a LoRA adapter switched on for questions and a tiny router, with MLX 8-bit builds available.

The tool is designed for local calibration and offline routing experiments, particularly for Mac indie developers.

Importance 1/3 Trust 1/3 Hugging Face Forums Alibaba (Qwen) Inference efficiency Open weights

Read original

media r/LocalLLaMA · 7d ago · 113 views

ISTA releases GSQ-RCO GGUFs for Qwen3.8-Flash-Next and 50% expert-pruned Coder build

The ISTA Deep Algorithms and Systems Lab has released quantized GGUF versions of the sparse mixture-of-experts model Qwen3.8-Flash-Next, alongside an experimental capability-targeted build with half its experts removed.

arxiv arXiv cs.LG · 43d ago · 71 views

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

The authors introduce Quantization-Aware Healing (QAH), a pipeline that distills 4-bit students directly from uncompressed models to recover performance lost during structural compression and quantization. Applied to GPT-OSS 120B, this method produces Hypernova-60B, which matches or beats the bfloat16 source on 7 of 9 benchmarks while using roughly 4 times less weight memory.

arxiv arXiv cs.CL · 43d ago · 75 views

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

The authors introduce Quantization-Aware Healing (QAH), a method to recover reasoning and coding capabilities in structurally compressed, 4-bit large language models. Unlike standard quantization-aware training which re-fits the compressed model, QAH distills the 4-bit student directly from the original uncompressed model.

media r/LocalLLaMA · 52d ago · 48 views

Qwen releases open weights for Qwen 3.8 27B

Qwen has released the open weights for its Qwen 3.8 27B model. The checkpoint is available on Hugging Face.

media MarkTechPost · 80d ago · 92 views

PrismML releases Bonsai 27B, compressing Qwen3.6-27B to 1-bit and ternary weights

PrismML has released Bonsai 27B, a low-bit quantization of the Qwen3.6-27B model that enables running large multimodal models on laptops and phones without retraining.