arxiv arXiv cs.AI · 46d ago · src: 51d ago · research

LLMs Benchmarked for Web Vulnerability Detection

from English

A study evaluates six LLMs on detecting real-world web vulnerabilities in WordPress plugins, finding detection rates vary by model and prompt design. Claude Opus 4.6 achieved the highest detection rate at 63%, while Qwen 3.5 only reached 35%, and no model consistently identified all baseline vulnerabilities across iterations.

Importance 2/3 arXiv cs.AI Mistral AI Alibaba (Qwen) xAI Code generation Evaluation & benchmarks Reasoning models

Read original

arxiv arXiv cs.AI · 46d ago · 1 view

Benchmark Evaluation of Small Language Models for Arabic NLP

A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.

arxiv arXiv cs.CL · 47d ago · 2 views

Benchmark Evaluation of Small Language Models for Arabic NLP

media r/LocalLLaMA · 52d ago · 2 views

Benchmarking small LLMs on hard HTML data extraction

A user tested models from 2B to 35B parameters on 29 difficult HTML data extraction pages, finding that smaller models like gemma4 e2b and e4b outperform larger ones. Qwen3.6 27B led in performance, while all MOE models scored poorly, highlighting the importance of task-specific benchmarking.

arxiv arXiv cs.CL · 47d ago · 1 view

P4IR Framework Improves LLM-Based Code Compliance Accuracy

P4IR, a two-stage framework, uses supervised fine-tuning and Group Relative Policy Optimization to enhance large language model-based automated code compliance systems. It reduces tree edit and token-level Levenshtein distances by up to 23.8% and 38.6% respectively, outperforming leading LLMs like Claude Opus, GPT-5.2, and GLM-4.7 in zero-shot settings with few-shot prompting, and reduces false positives by a statistically significant margin.

arxiv arXiv cs.CL · 47d ago · 6 views

Test-Time Steering Resolves Temporal Fact Conflicts in LLMs

Researchers identify parametric temporal conflicts in language models where outdated facts persist in parameters. They introduce Temporal Attractor Steering (TAS), a test-time method that resolves 29-57% of such conflicts without retraining, maintaining 85-99% accuracy on non-conflict queries and outperforming a baseline on three of four models.