A new study proposes that Rotary Position Embeddings (RoPE) frequencies are selected by models to match the relative-distance structure of their training data, rather than being used uniformly. The authors formalize a field-resolution tradeoff, showing that optimal frequency scales inversely with the width of the data-induced dependency profile.
- For a dependency profile of width W, the optimal RoPE frequency scales as 1/W.
- Mid-low frequency bands in language models arise from the multi-scale dependency structure of natural language.
- Scaling frequencies down expands the effective field but reduces resolution, aiding generalization when longer-context dependencies are approximate dilations of training dependencies.
- Natural language exhibits approximate self-similarity across positional scales, which explains why test-time frequency scaling supports long-context generalization.
The results identify a data-driven mechanism behind emergent RoPE usage and demonstrate that long-context generalization depends on scale matching between learned frequencies and how dependencies extend to longer contexts.