Hugging Face boosts tokenizers v1 for faster ML workflows

hugging face released tokenizers v1, a library for converting text into integers that models can read. the update focuses on performance, with improvements of tens of times over v0.23. key changes include a workspace split, making the library more modular, and a no-alloc model, reducing memory allocation. these changes aim to prevent gpu idleness in machine learning workflows. the library supports various tokenization algorithms, including byte pair encoding, wordpiece, and unigram.