arxiv arXiv cs.CL · 47d ago · src: 51d ago · research

Benchmark Evaluation of Small Language Models for Arabic NLP

from English

A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.

Importance 2/3 arXiv cs.CL Mistral AI Alibaba (Qwen) DeepSeek Code generation Evaluation & benchmarks Reasoning models

Read original

arxiv arXiv cs.AI · 46d ago · 1 view

Benchmark Evaluation of Small Language Models for Arabic NLP

arxiv arXiv cs.AI · 46d ago

LLMs Benchmarked for Web Vulnerability Detection

A study evaluates six LLMs on detecting real-world web vulnerabilities in WordPress plugins, finding detection rates vary by model and prompt design. Claude Opus 4.6 achieved the highest detection rate at 63%, while Qwen 3.5 only reached 35%, and no model consistently identified all baseline vulnerabilities across iterations.

media r/LocalLLaMA · 52d ago · 2 views

Benchmarking small LLMs on hard HTML data extraction

A user tested models from 2B to 35B parameters on 29 difficult HTML data extraction pages, finding that smaller models like gemma4 e2b and e4b outperform larger ones. Qwen3.6 27B led in performance, while all MOE models scored poorly, highlighting the importance of task-specific benchmarking.

arxiv arXiv cs.CL · 53d ago · 2 views

LLMs Predict Dementia and Depression from Clinical Speech

A study uses open-weight large language models to assess dementia and depression severity from clinical interviews. LLMs achieve accurate zero-shot depression prediction (MAE 0.60) and improved dementia assessment with feature extraction (MAE 0.78), reducing errors by up to 35%. Pause-enriched transcripts match human transcriptions, supporting automated screening pipelines for neuropsychiatric disorders.

arxiv arXiv cs.CL · 53d ago

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

A study challenges the assumption that visual attention signals reliability in vision-language models. It finds near-zero correlation between spatial attention and accuracy, showing instead that self-consistency across reasoning paths is a stronger predictor of truth. Reliability is better explained by generation dynamics and internal state distributions, not visual attention patterns.