A detailed analysis maps the KLD (Kullback-Leibler divergence) of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B models. Results show q8/q8 quantization is nearly lossless on both models, while q4/q4 performs well on Qwen but causes severe degradation on Gemma. Turbo quantization variants show mixed performance, with turbo3 and turbo2 enabling extreme cache compression at significant accuracy cost.
media
r/LocalLLaMA
·
66d ago
·
open_models
KLD Analysis of KV Cache Quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT
from English
Related articles
arxiv
arXiv cs.CL
·
50d ago
·
7 views
Distilling deepseek-r1:8B into Qwen3-0.6B enables fast on-device structured text enrichment
Researchers demonstrate that distilling an 8B reasoning teacher (deepseek-r1:8B) into a 0.6B student model (Qwen3-0.6B via QLoRA) allows for high-volume structured extraction with significantly lower latency and cost. The study evaluates this approach on mapping news articles to JSON objects containing summaries and categorical labels, comparing the distilled model against few-shot prompting and constrained decoding baselines.
media
r/LocalLLaMA
·
62d ago
·
1 view
SpectralQuant Qwen3.5 0.8B Q4_K_M recovers 96.5% of BF16 gap
Spectral Labs has released a release candidate for a calibration-aware Q4_K_M quantization of the Qwen3.5 0.8B model, utilizing a new method called SpectralQuant. This approach aims to make standard Q4_K_M footprints behave more like larger quant formats while maintaining compatibility with llama.cpp.
arxiv
arXiv cs.AI
·
64d ago
·
1 view
Automated Semantic Fault Localization in SysML v2 Using Knowledge-Graph Augmented LLMs
This paper presents a human-in-the-loop framework for automatically identifying and repairing semantic errors in SysML v2 models that compilers cannot detect. The approach combines fine-tuned Small Language Models with a domain knowledge graph to ground repair suggestions in valid engineering constraints.
arxiv
arXiv cs.AI
·
66d ago
·
5 views
Benchmark Evaluation of Small Language Models for Arabic NLP
A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.
arxiv
arXiv cs.CL
·
67d ago
·
5 views
Benchmark Evaluation of Small Language Models for Arabic NLP
A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.