The tokenizers library has released a version 1 release candidate that significantly improves performance for text encoding, addressing bottlenecks in training and serving workflows. The update maintains API compatibility with v0.23 while delivering substantial speedups across ten model families.
- Encoding is 3 to 30 times faster than v0.23 on a single thread by replacing regex splitting with bitstream operations via the bitcannon library.
- A thread-local word cache stores results for repeated pre-tokens, allowing subsequent occurrences to skip the merge process entirely.
- The BPE merge loop reuses scratch buffers and processes batches of pre-tokens in a single call, eliminating per-call memory allocations.
- Scaling across eight workers achieves 76% of linear performance, with decoding throughput also improved through direct buffer writes.
These optimizations ensure that tokenization no longer starves models of data by keeping GPUs from sitting idle while waiting for CPU processing.