A research preview of ThetaScan v0.1, a fixed-state nonlinear token mixer, has been released as an open-source implementation. The architecture ensures that every token computes its memory write from the current input and shared parameters rather than an evolving fast state, allowing writes to compose through an associative prefix scan.
The public repository includes several key components: ThetaScan-GN for local Jacobian or Gauss–Newton fast-weight updates, and ThetaScan-Kernel for normalized positive-feature numerator/mass memory. The implementation supports various kernel feature maps such as ReLU², thresholded ReLU², SiLU, and SwiGLU, along with normalized reads, EMA/recency banks, and deterministic random feature expansion.
Preliminary results from a single-seed, parameter-matched 17M language model experiment at step 7,500 show that the GN expanded 2× mixer achieved a raw bits per byte (BPB) of 1.2327 and an exact-int8 BPB of 1.23831. These early measurements are not claimed to be statistically superior due to differences in optimizer policies and the use of a quadratic reference evaluator rather than a fused linear-scaling implementation.
The authors welcome multi-seed replications, adversarial review of the write/read algebra, unified-optimizer comparisons, and long-context failure tests to further validate the approach.