media r/LocalLLaMA · 2d ago · open_models

ISTA releases GSQ-RCO GGUFs for Qwen3.8-Flash-Next and 50% expert-pruned Coder build

from English

The ISTA Deep Algorithms and Systems Lab has released quantized GGUF versions of the sparse mixture-of-experts model Qwen3.8-Flash-Next, alongside an experimental capability-targeted build with half its experts removed.

The release includes four quantized GGUFs ranging from 2.40 to 3.50 bpw (66.4 to 83.6 GB) using GSQ and RCO techniques.
At 3.50 bpw, the model matches the BF16 base on every benchmark evaluated, including GPQA-Diamond and LiveCodeBench v6.
The expert-pruned Coder build removes 256 of 512 routed experts per layer, resulting in an average bitwidth of 1.89 bits per parameter.
This pruning reduces the resident working set to 29.6 GB, allowing the 176.9B-parameter model to fit within a single 32 GB accelerator.

The quantized models maintain high performance relative to their size, while the pruned Coder build retains 91.3% of SWE-bench Verified and 98.7% of LiveCodeBench v6 scores compared to the BF16 baseline.

Importance 2/3 r/LocalLLaMA Alibaba (Qwen) Inference efficiency Open weights

Benchmarks

Benchmark	Model	Score
GPQA Diamond	Qwen3.8-Flash-Next (IQ3_XXS)	91.41%
LiveCodeBench	Qwen3.8-Flash-Next (IQ3_S)	86.86%
LiveCodeBench	Qwen3.8-Flash-Next (IQ3_XXS)	86.29%
LiveCodeBench	Qwen3.8-Flash-Next Coder	86.28%
SWE-bench Verified	Qwen3.8-Flash-Next Coder	75.6%

Read original

arxiv arXiv cs.LG · 38d ago · 47 views

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

The authors introduce Quantization-Aware Healing (QAH), a pipeline that distills 4-bit students directly from uncompressed models to recover performance lost during structural compression and quantization. Applied to GPT-OSS 120B, this method produces Hypernova-60B, which matches or beats the bfloat16 source on 7 of 9 benchmarks while using roughly 4 times less weight memory.

arxiv arXiv cs.CL · 38d ago · 49 views

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

The authors introduce Quantization-Aware Healing (QAH), a method to recover reasoning and coding capabilities in structurally compressed, 4-bit large language models. Unlike standard quantization-aware training which re-fits the compressed model, QAH distills the 4-bit student directly from the original uncompressed model.

media r/LocalLLaMA · 47d ago · 27 views

Benchmarks

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

Quantization-Aware Healing recovers 4-bit LLMs faster than QAT

Qwen releases open weights for Qwen 3.8 27B

PrismML releases Bonsai 27B, compressing Qwen3.6-27B to 1-bit and ternary weights

Qwen3.5 122B A10B at UD-Q2_K_XL fits in 64GB RAM and improves internal knowledge