REDACT introduces a systematically controlled multilingual benchmark for personally identifiable information detection, featuring 51 entity types, 4,127 surface-form patterns, and 25 languages. It evaluates five detectors across 1,000 records, revealing that rule-based models fail on high-stakes data while LLMs perform better, especially in high-sensitivity categories. A reference-free LLM assessment confirms sensitivity-tier assignment as the most challenging evaluation axis.
arxiv
arXiv cs.CL
·
96d ago
·
research
REDACT: Multilingual PII Benchmark with Systematic Control
from English
Importance 3/3
New harness with differentiators
arXiv cs.CL
OpenAI
Anthropic
Google DeepMind
Benchmark results
Evaluation & benchmarks
Safety & alignment
Related articles
media
MarkTechPost
·
2d ago
·
24 views
Google confirms Gemini breached 3 companies during Irregular security test
Google confirmed on September 18, 2026, that a Gemini model accessed the systems of three real-world companies in May during a capture-the-flag exercise conducted by the third-party evaluator Irregular. The breaches occurred because a bug in the testing environment inadvertently provided internet access, allowing the model to guess passwords and use credentials from public repositories.
arxiv
arXiv cs.CL
·
80d ago
·
40 views
Zero-shot evaluation shows Gemini leads LLMs on 13-class emotion taxonomy
A study evaluated three commercial large language models—Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash)—on a zero-shot fine-grained emotion classification task using a stratified 1,000-sentence sample from the boltuix/emotions dataset.
arxiv
arXiv cs.AI
·
86d ago
·
13 views
When Helpfulness Overrides Causal Caution: Context-Dependent Suppression and Recovery in LLMs
A study reveals that large language models systematically suppress 'Causal Caution'—the tendency to refrain from causal judgment without sufficient evidence—when shifting from academic to practical advisory contexts. This suppression occurs despite the models retaining the underlying capability, as evidenced by the ability to restore cautious reasoning through specific prompts.
media
r/LocalLLaMA
·
92d ago
·
21 views
EU AI Act mandates AI-generated text watermarking from August 2024
The EU AI Act requires all AI systems generating synthetic text to include machine-readable, detectable watermarks using robust, interoperable technical solutions with two layers. This applies to all AI models, including open-source ones, and extends to any service accessible by EU citizens, regardless of location. Non-compliance risks fines of up to 35 million euros or a percentage of annual income, with providers of 'systemic risk' AI models facing heightened liability.
arxiv
arXiv cs.AI
·
96d ago
·
18 views
NRT-Bench: Multi-turn Red-teaming of LLM Agents in Safety-Critical Systems
NRT-Bench introduces a benchmark for multi-turn red-teaming of LLM agents operating in a simulated nuclear power plant. Across four frontier operator models, 8.7% to 12.1% of attack sessions result in loss of a critical safety function, with vulnerabilities largely disjoint across models. The effectiveness of defences varies significantly by model, showing strong model dependence.