This paper challenges the assumption that language models must be trained on text-only representations by demonstrating that unsupervised visual pretraining is a scalable approach for foundation model intelligence.
The authors conduct a systematic study of visual pretraining paradigms that directly leverage visual documents without text extraction. Across multiple backbones and benchmarks, visual pretraining on the same underlying corpora consistently outperforms text-only pretraining.
This work offers an efficient pathway to scalable language intelligence by retaining rich information from figures, typeset equations, and page layouts that is lost when converting sources to plain text.