A controlled comparison of byte-pair encoding (BPE) and Unigram-LM tokenizers on a fixed 165-token chemistry base reveals that the two algorithms produce structurally different subword vocabularies. Across three corpus typologies and various boundary policies, the learned pieces show a cross-algorithm Jaccard overlap never exceeding 0.161.
- Unigram-LM segments held-out molecules into 29-41% more tokens than BPE.
- The algorithms largely agree on where to cut but differ in depth, with BPE acting as a strict coarsening of Unigram-LM's segmentation on 80-99% of molecules.
- This separation persists across corpus types, boundary policies, and vocabulary sizes, even at eight times the original scale.
The study concludes that the choice of subword algorithm is a critical modeling decision rather than a free default for chemical language models.