Procrustes-conditioned Joint End-to-end Top-K SAE extracts universal features from BERT

The authors present a Procrustes-conditioned Joint End-to-end Top-K Sparse Autoencoder (SAE) designed to extract cross-seed universal features from independently trained BERT models. This approach addresses the challenge of misaligned feature spaces caused by non-convex dictionary learning and random initialization.

• The method computes an orthogonal Procrustes rotation between seeds' activation spaces before joint SAE training.
• It combines Top-K sparsity, end-to-end downstream optimization, and an auxiliary dead-feature revival loss.
• Evaluation on five independent seed pairs (ten BERT models) across SST-2, Stanford Politeness, and TweetEval Emotion shows Pearson r ≥ 0.70 across seeds.
• The pipeline produces more universal features than post-hoc alignment baselines on all three datasets.

The authors consider this important because it enables the extraction of interpretable sociolinguistic patterns that are consistent across different model initializations.