arxiv arXiv cs.LG · 53d ago · research

Baseline Evaluation of Open-Source LLMs for Multi-Label ATT&CK Classification

from English

A ground-truth dataset of 2,076 human-annotated sentences from 83 complex CTI reports was constructed and mapped to 114 ATT&CK techniques with \k{appa} = 0.68 inter-annotator agreement. Seven open-source LLMs ranging from 8B to 236B parameters were evaluated, achieving a maximum micro-averaged F1 score of 0.22. Parameter size showed a statistically significant positive correlation with F1 score, while prompt strategy and temperature did not yield significant improvements, indicating current open-source LLMs are insufficient for production-grade ATT&CK classification.

Importance 2/3 arXiv cs.LG OpenAI DeepSeek Mistral AI Evaluation & benchmarks Open weights Reasoning models

Read original

arxiv arXiv cs.AI · 52d ago · 1 view

X+Slides: Benchmark for Audience-Conditioned Slide Generation

X+Slides introduces a benchmark that evaluates slide generation based on target audience needs. It uses 8,133 source-grounded probes across 113 topics and seven scenes to measure Audience Coverage, Domain-wise Coverage, Efficiency, and Correctness, revealing that current systems recover only partial audience-essential information, with DeepPresenter achieving 0.714 Audience Coverage, SlideTailor 0.594, and NotebookLM ablation 0.853, highlighting the need for source-grounded evaluation.

arxiv arXiv cs.CL · 53d ago · 2 views

LLMs Predict Dementia and Depression from Clinical Speech

A study uses open-weight large language models to assess dementia and depression severity from clinical interviews. LLMs achieve accurate zero-shot depression prediction (MAE 0.60) and improved dementia assessment with feature extraction (MAE 0.78), reducing errors by up to 35%. Pause-enriched transcripts match human transcriptions, supporting automated screening pipelines for neuropsychiatric disorders.

arxiv arXiv cs.CL · 41d ago

Triadic Werewolf: A Jester Role for Multi-Hop Theory of Mind in LLMs

Researchers extended the Werewolf game with a Jester role to create a triadic social-deduction environment that requires reasoning across three opposing utility functions, challenging large language models' theory-of-mind capabilities. Evaluations on GPT-4.1, DeepSeek-V3.1, and Llama-3.3-70B revealed that while the Jester won 60-70% of games, GPT-4.1 wolves frequently voted the Jester out on day 1 in 60-70% of cases, a self-defeating action driven by language priors.

arxiv arXiv cs.AI · 46d ago · 1 view

Benchmark Evaluation of Small Language Models for Arabic NLP

A benchmark of 240 Arabic test items across eight domains and ten skills assesses twelve small language models in zero-shot settings. Gemma 3 (12B) achieved the highest overall score (4.548/5), followed by Aya and C4AI Command Arabic, with performance linked more to Arabic alignment and instruction-following than model size. Common failure modes include prompt leakage, hallucination, and weak task adherence.

arxiv arXiv cs.CL · 47d ago · 6 views

Test-Time Steering Resolves Temporal Fact Conflicts in LLMs

Researchers identify parametric temporal conflicts in language models where outdated facts persist in parameters. They introduce Temporal Attractor Steering (TAS), a test-time method that resolves 29-57% of such conflicts without retraining, maintaining 85-99% accuracy on non-conflict queries and outperforming a baseline on three of four models.