media Hugging Face Forums · just now · open_models

ByteLex uses tokenizer map to predict and fix byte-model errors

from English

ByteLex researchers built a coordinate space from 11 tokenizer vocabularies to predict which made-up words their byte-level language model would fail on. The map achieved a Spearman correlation of -0.98 with the model's measured per-word accuracy, outperforming corpus-derived statistics.

The model is a 237M-parameter byte-level language model trained on raw UTF-8 without tokenization.
Errors were found to depend primarily on the first byte, where English byte-priors hijack the answer.
The map was used to generate targeted training data for predicted weak spots.
Held-out accuracy improved from .53/.40 to .84/.79 after applying the fix.

This approach allows developers to identify and correct specific failure modes in tokenizer-free models using only structural analysis of existing tokenizers.

Importance 1/3 Trust 1/3 Hugging Face Forums Evaluation & benchmarks Training methods

Read original

media Hugging Face Forums · 23d ago · 28 views

metrictrust detects when validation metrics are untrustworthy during training

The author developed metrictrust, a lightweight Python tool that monitors training runs to determine if final validation metrics should be trusted. It addresses common issues such as keeping suboptimal checkpoints, overestimating performance due to memorization, and relying on noisy oscillating results.

arxiv arXiv cs.CL · 63d ago · 13 views

CARE-PPO enables joint learning of accurate predictions and reliable confidence in language-based quantitative tasks

Researchers introduce CARE-PPO, a reinforcement learning framework that connects loss prediction for uncertainty estimation with actor-critic PPO fine-tuning. This approach allows for the joint learning of accurate numerical estimates and reliable confidence signals in language-based quantitative prediction from unstructured inputs.

arxiv arXiv cs.CL · 78d ago · 11 views

REAR: Test-time Preference Realignment through Reward Decomposition

The authors introduce REAR, a novel framework that extends test-time scaling (TTS) to preference alignment by modeling the task as a realignment problem. This approach addresses the limitation of existing TTS methods, which are typically restricted to verifiable domains like mathematics and coding.

arxiv arXiv cs.CL · 78d ago · 4 views

The Hidden Cost of Resampling: How Imbalance Correction Degrades Probability Calibration in Tree Ensembles

This study evaluates the impact of resampling methods like SMOTE and random undersampling on probability calibration in tree ensembles, finding that while SMOTE's cost is small, undersampling severely degrades calibration.

arxiv arXiv cs.CL · 78d ago · 5 views

Pre-Registered Screening Rule for Evolutionary Outer Loops

The authors introduce a pre-registered screening rule that determines before implementation whether an evolutionary outer loop over neural network parameters is worth building compared to a cheap single-shot alternative. The rule calculates a recovery metric R, defined as the best single-shot gain divided by the best gain of any cheap method, and prescribes skipping the outer loop when R is greater than or equal to 90%.