Signed-permutation gauge fixes coordinate transport for RMSNorm Transformers

The article demonstrates that coordinate-indexed objects in LLM workflows require fixing the model's residual-stream gauge, which is architecture-dependent. While LayerNorm models have a permutation gauge ($S_d$), RMSNorm models possess a signed-permutation gauge ($B_d$), making permutation-only alignment symmetry-incomplete.

• The authors introduce sign-marginalized Hungarian matching to address structural accuracy ceilings in raw signed-correlation matching.
• Composing saved-checkpoint local $B_d$ gauges recovers 91.1% of cross-run coordinates at 1500 steps, compared to 60.3% for endpoint matching.
• Under the $B_d$ gauge, TinyLlama SAE reconstruction achieves an NMSE of 0.004 versus 1.08 under $S_d$, and Qwen sentiment steering preserves 95.8% of its effect versus 17.2%.
• Signed transport of AdamW state preserves the resumed training trajectory, whereas permutation-only state follows a different one.

The authors argue that coordinate-preserving transport is essential for tools like SAEs and steering vectors that break under permutation-only alignment, and that interpretability claims are only reproducible relative to an explicit gauge.