A study demonstrates that FFT-based spectral preprocessing of learned query-key (Q/K) projections substantially improves transformer attention on character-level language modelling.
- On TinyShakespeare, a fixed random spectral filter achieves val=1.031, while four learned frequencies spanning paragraph to word scale achieve val=0.309, representing a 79% reduction over standard dot-product attention.
- The single-frequency result is confirmed across three random seeds with a mean val of 0.236 and std of 0.019.
- The four frequencies converge to a near-geometric multi-scale ordering (49, 27, 10, 6 tokens/cycle) corresponding to paragraph, sub-paragraph, phrase, and word scales.
- Gains are specific to spectral preprocessing; random orthogonal and non-orthogonal projections produce no measurable improvement.
- Causal filters do not improve over standard attention at character-level tokenisation, as the bilateral FFT kernel is structurally non-causal.
This work defines an architectural boundary between bilateral spectral attention and genuinely causal spectral attention, distinct from FNet which replaces attention with Fourier mixing of token embeddings.