The article investigates whether an equivalent tokenizer for a closed-source large language model can be reconstructed using only two oracles derived from its chat API. These oracles are the token length oracle, which returns the number of tokens in a string, and the prefix token oracle, which decodes the first n tokens to resolve merge order ambiguities.
- Token length oracle: Given any string s, it returns len(tokenize(s)).
- Prefix token oracle: Given a string s and integer n, it returns the string decoded from the first n tokens of tokenize(s).
- The prefix token oracle is constructed by leveraging the model's ability to repeat text in a controlled way via the regular chat interface.
The core question posed is whether these two oracles are sufficient to reconstruct an equivalent tokenizer that produces exact same token boundaries on any input, up to token ID remapping.