media Hugging Face Forums · 1h ago · open_models

metrictrust detects when validation metrics are untrustworthy during training

from English

The author developed metrictrust, a lightweight Python tool that monitors training runs to determine if final validation metrics should be trusted. It addresses common issues such as keeping suboptimal checkpoints, overestimating performance due to memorization, and relying on noisy oscillating results.

Provides a plain-language verdict of 'trust', 'caution', or 'do-not-trust' based on training dynamics.
Detects when the best metric occurred mid-training but was not saved as the final checkpoint.
Identifies cases where train accuracy significantly exceeds eval accuracy, indicating memorization.
Flags runs with high oscillation to warn against treating a single noisy sample as a reliable measurement.
Works with HuggingFace Trainer via TrustCheckCallback or any custom loop via TrustMonitor.

The tool helps users avoid drawing incorrect conclusions from final log numbers by highlighting when the model's peak performance was missed or when the data is too unstable to be meaningful.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Training methods

Read original

arxiv arXiv cs.CL · 40d ago · 14 views

CARE-PPO enables joint learning of accurate predictions and reliable confidence in language-based quantitative tasks

Researchers introduce CARE-PPO, a reinforcement learning framework that connects loss prediction for uncertainty estimation with actor-critic PPO fine-tuning. This approach allows for the joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction from unstructured inputs.

arxiv arXiv cs.CL · 55d ago · 1 view

REAR: Test-time Preference Realignment through Reward Decomposition

The authors introduce REAR, a novel framework that extends test-time scaling (TTS) to preference alignment by modeling the task as a realignment problem. This approach addresses the limitation of existing TTS methods, which are typically restricted to verifiable domains like mathematics and coding.

arxiv arXiv cs.CL · 55d ago

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

This study evaluates the impact of resampling methods like SMOTE and random undersampling on probability calibration in tree ensembles, finding that while SMOTE's cost is small, undersampling severely degrades calibration.

arxiv arXiv cs.CL · 56d ago

Pre-Registered Screening Rule for Evolutionary Outer Loops

The authors introduce a pre-registered screening rule that determines before implementation whether an evolutionary outer loop over neural network parameters is worth building compared to a cheap single-shot alternative. The rule calculates a recovery metric R, defined as the best single-shot gain divided by the best gain of any cheap method, and prescribes skipping the outer loop when R is greater than or equal to 90%.

arxiv arXiv cs.CL · 56d ago · 5 views

DistilledGemma: Balanced Efficiency-Accuracy for Person-Place Relation Extraction

The authors present DistilledGemma, an efficient system for person-place relation extraction from multilingual historical newspaper articles in English, German, and French. The approach utilizes a three-stage knowledge distillation pipeline to balance classification accuracy with computational efficiency.