TGO-II analyzes how ViT representation geometry evolves during supervised training

The Transformer Geometry Observatory-II (TGO-II) is a framework designed to investigate the evolution of Vision Transformer representations during supervised training, addressing the lack of understanding regarding their geometric changes.

• TGO-II analyzes ViT-Small/16 representations using Centered Kernel Alignment (CKA), Singular Vector Canonical Correlation Analysis (SVCCA), Two-Nearest Neighbor Intrinsic Dimensionality (TwoNN-ID), and token covariance analysis.
• CKA and SVCCA scores progressively decrease throughout training, indicating increasing representational specialization across layers.
• Intrinsic dimensionality consistently increases before stabilizing, suggesting the representation manifold expands into a larger set of locally accessible degrees of freedom.
• Token covariance and coupling analyses show that strong token interaction structure persists throughout training.

These findings suggest that representation complexity and layer specialization emerge simultaneously during training, with manifold expansion occurring without token decoupling.