This study evaluates whether fine-tuned ModernBERT encoder classifiers can serve as cost-effective alternatives to LLM-based judges for safety evaluation. The researchers benchmarked ModernBERT and Ettin against rule-based prefix matching, fine-tuned LLM classifiers, and various LLM judge methodologies. These LLM judges included strategies from StrongReject, ShieldGemma, JailbreakBench, AILuminate, SorryBench, Claude-as-a-judge, and models like LlamaGuard 3 and 4. The encoder classifiers were trained on judge-labeled data using a majority-voting label strategy and tested on a gold-standard holdout dataset. Performance was measured using F1 score, false negative rate, and precision-recall metrics across open-source adversarial datasets. Results were further analyzed by attack technique, including single-turn prompting, decomposition, escalation, and context manipulation. The findings provide guidance on when encoder classifiers can reliably replace LLM-based judges without substantial performance loss.
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation
LLMs show potential but need improvement for antisemitic incident classification
This study evaluates the ability of large language models to detect and classify reports of antisemitic events using fine-grained labels. The authors tested OpenAI's GPT-4o and Meta's Llama-3.2-3B-Instruct on expert-annotated datasets derived from news articles, civil society reports, and official records.
PSALM framework evaluates LLM stylistic appropriation under EU copyright law
The authors introduce PSALM, an LLM-as-a-judge framework designed to operationalize EU copyright doctrine for evaluating stylistic appropriation in large language models. Unlike existing safeguards that focus on verbatim memorization, PSALM assesses computational overlap and stylistic dimensions such as writing style and narrative voice.
Calibration Without Comprehension in LLM Vulnerability Detection
CWE-Trace evaluates eight vanilla and 15 LoRA-fine-tuned LLMs on Linux kernel vulnerability detection. Results show data contamination offers no advantage, and fine-tuning only shifts output thresholds without altering decision policies. Despite improved detection scores, LLMs lack reliable security reasoning, with top-1 CWE accuracy below 1.3% and binary detection performance at 52.1%.
LLM Alignment Using Implicit User Feedback
A new dataset, IFLLM, collects mouse trajectories and eye gazing data from users interacting with LLMs. It shows that implicit feedback significantly improves LLM alignment, boosting text-based reward model accuracy from 55% to 64% and nearly tripling response quality improvements after DPO training on eight LLMs.
LLM Alignment Using Implicit User Feedback
A new dataset, IFLLM, collects mouse trajectories and eye gazing data from users interacting with LLMs. It shows that implicit feedback significantly improves LLM alignment, boosting text-based reward model accuracy from 55% to 64% and nearly tripling response quality improvements after DPO training on eight LLMs.