REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.
arxiv
arXiv cs.CL
·
51d ago
·
research
REDACT: Multilingual PII Benchmark with Systematic Control
from English
Importance 3/3
New harness with differentiators
arXiv cs.CL
OpenAI
Anthropic
Google DeepMind
Benchmark results
Evaluation & benchmarks
Safety & alignment
Related articles
arxiv
arXiv cs.CL
·
35d ago
Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy
A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.
arxiv
arXiv cs.AI
·
40d ago
·
1 view
When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs
A study reveals that large language models systematically suppress 'Causal Caution'—the tendency to refrain from causal judgment without sufficient evidence—when shifting from academic to practical advisory contexts. This suppression occurs despite the models retaining the underlying capability, as evidenced by the ability to restore cautious reasoning through specific prompts.
media
r/LocalLLaMA
·
47d ago
·
4 views
EU AI Act mandates AI-generated text watermarking from August 2024
The EU AI Act requires all AI systems generating synthetic text to include machine-readable, detectable watermarks using robust, interoperable technical solutions with two layers. This applies to all AI models, including open-source ones, and extends to any service accessible by EU citizens, regardless of location. Non-compliance risks fines of up to 35 million euros or a percentage of annual income, with providers of 'systemic risk' AI models facing heightened liability.
arxiv
arXiv cs.AI
·
51d ago
·
5 views
NRT-Bench: Multi-turn Red-teaming of LLM Agents in Safety-Critical Systems
NRT-Bench introduces a benchmark for multi-turn red-teaming of LLM agents operating in a simulated nuclear power plant. Across four frontier operator models, 8.7% to 12.1% of attack sessions result in loss of a critical safety function, with vulnerabilities largely disjoint across models. The effectiveness of defences varies significantly by model, showing strong model dependence.
arxiv
arXiv cs.AI
·
51d ago
·
2 views
Defensive Misdirection Against Automated Attacks on Agentic AI
Agentic AI systems face growing threats from model-guided automated attacks. A new defense strategy, Contextual Misdirection via Progressive Engagement (CMPE), reduces attacker success rates by up to two orders of magnitude and nearly eliminates verified attack success in benchmark tests.