arxiv arXiv cs.LG · 15h ago · research

Pangram Labs finds wild AI text harms pretraining and proposes new scaling law

from English

A Pangram Labs study of "wild" AI-generated web text reveals that adding such tokens to pretraining data initially lowers loss on human text but quickly reverses into harm as the ratio increases. The researchers pretrained 800 language models to derive a new scaling law that accounts for this dynamic, which existing laws like Chinchilla fail to predict.

Analysis of FineWeb data shows AI-generated tokens rose from 27.5% in June 2026 to 31.1% by August.
For data-starved models, added AI tokens lower human text loss initially but saturate and then increase it.
Models trained on high budgets of human text see their loss rise almost immediately upon adding AI tokens.
The new scaling law predicts held-out human-text loss for models up to 3.6x larger with 41% lower error than existing laws.

The authors recommend filtering AI text when the target is human text, repeating human text before expanding datasets with AI-generated content, and reporting validation loss on human and AI text separately.

Importance 2/3 arXiv cs.LG Research paper Training data

Related articles

arxiv arXiv cs.CL · 18h ago · 2 views

Pangram Labs finds wild AI text harms pretraining and proposes new scaling law

A Pangram Labs study of 800 pre-trained language models reveals that "wild" AI-generated web text, which now comprises over 31% of recent web data, negatively impacts model performance when the goal is to process human text. The research demonstrates that while small amounts of AI tokens may initially help data-starved models, they quickly reverse into harm as their proportion increases.

arxiv arXiv cs.LG · 78d ago · 19 views

CSFS reduces wind and solar power prediction feature selection cost by 21%

Researchers propose Cluster-based Sequential Feature Selection (CSFS), a novel clustering-based wrapper method for automatic and efficient feature selection in renewable energy prediction pipelines. The approach addresses the lack of systematic methods for selecting input features from the large number of available monitoring and environmental variables.

arxiv arXiv cs.AI · 78d ago · 18 views

CSFS reduces computational cost by 21% in renewable energy prediction

Researchers propose Cluster-based Sequential Feature Selection (CSFS), a novel, model-agnostic wrapper method designed to address the lack of systematic feature selection in wind and solar power prediction. The approach aims to provide automatic, efficient, and reliable feature selection for renewable energy pipelines.

arxiv arXiv cs.AI · 78d ago · 16 views

FormalAnalyticGeo generates multimodal analytic geometry problems via neural-symbolic pipeline

Researchers introduce FormalAnalyticGeo, a scalable framework for the fully automatic generation of multimodal analytic geometry problems that eliminates the need for human annotation. The system utilizes a formal intermediate representation called CDL to bridge problem text with precise diagram rendering through a Signed Distance Field engine.

arxiv arXiv cs.CL · 80d ago · 16 views

Study finds small negative trend in German and English compound compositionality over time

Researchers investigate semantic change in 23 German and 26 English noun compounds by introducing the Compositionality Trend Prediction task, evaluated against a novel dataset of per-decade diachronic ratings. Contrary to literature positing a decisive tendency for compounds to become less compositional, the study finds only a small negative trend over time.