Researchers propose Riemannian Mean Pooling (RMP), a method that extracts per-token pullback metrics from an encoder's Jacobian and aggregates them using the Fréchet mean on the symmetric positive definite manifold. This approach aims to understand whether sentence-level classification signal resides in the Riemannian geometry of contextual token embeddings.
- RMP outperforms Euclidean mean pooling across three datasets with non-trivial linguistic structure: CoLA, CREAK, and RTE.
- On FEVER-Symmetric, a benchmark designed to remove annotation-driven lexical artifacts, the method correctly stays at chance.
- Ablations indicate that random initialization combined with Fréchet aggregation beats Euclidean pooling on two of the three signal-bearing datasets.
- The trained encoder contributes additional signal specifically on CREAK, which is identified as the most knowledge-heavy dataset among the three.
The study localizes the performance gain to the geometric aggregation method rather than learned manifold structure, demonstrating that Riemannian geometry effectively captures classification signals in pre-trained language model embeddings.