arxiv arXiv cs.CL · 71d ago · research

Selective Verification for Budget-Aware Reasoning

from English

Sevra, a serving-layer controller, selectively verifies answers to improve accuracy and reduce token usage. On \mathfive, it achieves 76.3% accuracy with 26.8% fewer post-generation tokens and halved harmful flips, while on \gsm it verifies only 3.0% of examples, boosting accuracy to 94.5% and cutting verification tokens by 91.2%. The study shows that initial solve length and explicit control needs determine optimal verification strategy.

Importance 3/3 New feature vs. leaders New harness with differentiators arXiv cs.CL Alibaba (Qwen) Evaluation & benchmarks Inference efficiency Reasoning models

Read original

arxiv arXiv cs.CL · 44d ago · 14 views

GSM-Plus-BN introduces perturbation-based benchmark for Bangla math reasoning

The study introduces GSM-Plus-BN, a novel perturbed Bengali mathematical dataset derived from the English GSM-Plus benchmark and verified by human translators. This resource addresses the lack of systematic benchmarks for assessing model robustness in linguistically diverse regions like Bangladesh.

media r/LocalLLaMA · 62d ago · 1 view

SpectralQuant Qwen3.5 0.8B Q4_K_M recovers 96.5% of BF16 gap

Spectral Labs has released a release candidate for a calibration-aware Q4_K_M quantization of the Qwen3.5 0.8B model, utilizing a new method called SpectralQuant. This approach aims to make standard Q4_K_M footprints behave more like larger quant formats while maintaining compatibility with llama.cpp.

arxiv arXiv cs.AI · 66d ago · 1 view

LLMs Benchmarked for Web Vulnerability Detection

A study evaluates six LLMs on detecting real-world web vulnerabilities in WordPress plugins, finding detection rates vary by model and prompt design. Claude Opus 4.6 achieved the highest detection rate at 63%, while Qwen 3.5 only reached 35%, and no model consistently identified all baseline vulnerabilities across iterations.

arxiv arXiv cs.AI · 66d ago · 5 views

Benchmark Evaluation of Small Language Models for Arabic NLP

A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.

media r/LocalLLaMA · 66d ago · 3 views

KLD Analysis of KV Cache Quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT

A detailed analysis maps the KLD (Kullback-Leibler divergence) of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B models. Results show q8/q8 quantization is nearly lossless on both models, while q4/q4 performs well on Qwen but causes severe degradation on Gemma. Turbo quantization variants show mixed performance, with turbo3 and turbo2 enabling extreme cache compression at significant accuracy cost.