tokenizers v1: encode, decode and scaling, measured

A tokenizer converts text into sequences of integers that machine learning models can read. The upcoming version 1 of the tokenizers library focuses on making this conversion process significantly faster than the previous version, with improvements often measured in tens of times speedup. As models become faster and workloads scale, tokenization can become a bottleneck that prevents GPUs from being fully utilized, making performance improvements necessary. The speedups come from multiple changes including using bitstream operations instead of regex patterns for splitting text, caching repeated words to avoid reprocessing them, rewriting the merging loop to update indices instead of moving data, and enabling multiple threads to tokenize simultaneously without queuing on a single lock.