The article argues that within-class variance in language-model representations is not incomplete neural collapse but rather allocated information storage governed by a specific law. It demonstrates that macro-category structure accounts for only 4-12% of representational variance, while within-token context carries 79-91%, a ratio stable across a 100x parameter range.
- A centering identity invalidates simplex equiangular-tight-frame claims regarding representation structure.
- Token-level weight decay reduces next-token prediction to an imbalanced K-class problem where category norms are ordered by type count.
- Within-category dispersion is forced to be proportional to the conditional mutual information I(token; context | category).
- Identity dispersion tracks this information across every tested model and partition, with one model's information predicting another's dispersion.
The findings suggest that the information carried by categories never leaves during pretraining, even as its relative share overshoots, decays, and partially recovers.