The article argues that within-class variance in language model representations is not incomplete neural collapse but rather allocated information storage governed by a specific law. Analysis of 14 models shows that macro-category structure accounts for only 4-12% of representational variance, while within-token context carries 79-91%, remaining stable across a 100x parameter range.
- Token-level weight decay penalizes categories by type count rather than occurrence mass, reducing next-token prediction to an imbalanced K-class problem where category norms are ordered by type count.
- A converse floor for binary categories forces within-category dispersion to be at least proportional to the conditional mutual information I(token; context | category).
- Identity dispersion tracks this information across every tested model and partition, with one model's information predicting another's dispersion under a model-free estimate.
The authors conclude that the category share overshoots, decays, and partially recovers over pretraining because the information it must carry never leaves the representation.