arxiv arXiv cs.CL · 18h ago · research

Pangram Labs finds wild AI text harms pretraining and proposes new scaling law

from English

A Pangram Labs study of 800 pre-trained language models reveals that "wild" AI-generated web text, which now comprises over 31% of recent web data, negatively impacts model performance when the goal is to process human text. The research demonstrates that while small amounts of AI tokens may initially help data-starved models, they quickly reverse into harm as their proportion increases.

For models trained on high budgets of human text, adding AI tokens raises loss almost immediately, whereas fresh human tokens continue to lower it.
Existing scaling laws fail to predict this behavior; the authors propose a new law with separate benefit and harm terms that reduces prediction error by 41% compared to existing methods.
The study recommends filtering AI text for human-targeted models, repeating human text before adding AI data, and reporting validation loss on human and AI text separately.

The findings suggest that AI text remains valuable only when the target is AI text, but should be filtered or handled carefully when training for human-centric tasks. The authors release WildAI, an 83B-token corpus with labels, along with code and model weights.

Importance 2/3 arXiv cs.CL Research paper Training data

Related articles

arxiv arXiv cs.LG · 15h ago · 1 view

Pangram Labs finds wild AI text harms pretraining and proposes new scaling law

A Pangram Labs study of "wild" AI-generated web text reveals that adding such tokens to pretraining data initially lowers loss on human text but quickly reverses into harm as the ratio increases. The researchers pretrained 800 language models to derive a new scaling law that accounts for this dynamic, which existing laws like Chinchilla fail to predict.

arxiv arXiv cs.LG · 78d ago · 19 views

CSFS reduces wind and solar power prediction feature selection cost by 21%

Researchers propose Cluster-based Sequential Feature Selection (CSFS), a novel clustering-based wrapper method for automatic and efficient feature selection in renewable energy prediction pipelines. The approach addresses the lack of systematic methods for selecting input features from the large number of available monitoring and environmental variables.

arxiv arXiv cs.AI · 78d ago · 18 views

CSFS reduces computational cost by 21% in renewable energy prediction

Researchers propose Cluster-based Sequential Feature Selection (CSFS), a novel, model-agnostic wrapper method designed to address the lack of systematic feature selection in wind and solar power prediction. The approach aims to provide automatic, efficient, and reliable feature selection for renewable energy pipelines.

arxiv arXiv cs.AI · 78d ago · 16 views

FormalAnalyticGeo generates multimodal analytic geometry problems via neural-symbolic pipeline

Researchers introduce FormalAnalyticGeo, a scalable framework for the fully automatic generation of multimodal analytic geometry problems that eliminates the need for human annotation. The system utilizes a formal intermediate representation called CDL to bridge problem text with precise diagram rendering through a Signed Distance Field engine.

arxiv arXiv cs.CL · 80d ago · 16 views

Study finds small negative trend in German and English compound compositionality over time

Researchers investigate semantic change in 23 German and 26 English noun compounds by introducing the Compositionality Trend Prediction task, evaluated against a novel dataset of per-decade diachronic ratings. Contrary to literature positing a decisive tendency for compounds to become less compositional, the study finds only a small negative trend over time.