tokenizers v1 release-candidate benchmarks show major encode/decode speedups
The article presents benchmarks and engineering changes in the release candidate of tokenizers v1, showing often-tens× speedups versus v0.23 and comparing to other tokenization libraries via the tokbench suite. The v1 refactor preserves token IDs, API, vocabulary and model support while introducing SIMD-friendly hand-written pre-tokenization, model-stage optimizations, multithreading and broader platform testing with contributors such as IBM and NVIDIA.