Executive brief
Hugging Face tokenizers is a library that performs text tokenization for machine learning models. A malicious tokenizer.json configuration file can trigger a buffer overflow when loaded, causing the application to crash immediately—even before any text is processed. An attacker could exploit this to cause denial of service in any service that loads untrusted tokenizer configurations.
Technical details
The vulnerability is an out-of-bounds buffer access in BpeBuilder::build (tokenizers/src/models/bpe/model.rs) that occurs during tokenizer deserialization. When loading a tokenizer.json file, the code allocates a scratch buffer sized to the longest vocabulary key, then writes merge rules into it. If a merge rule's concatenated token exceeds the longest vocabulary key size, the write overruns the buffer, triggering a Rust panic that aborts the process. Attack vector is local/adjacent through crafted tokenizer.json files; no authentication or encoding operations are required. A secondary defect can cause usize underflow when continuing_subword_prefix is configured and a merge token is shorter than the prefix. Observed in version 0.23.1; patch availability in later versions is indicated by references to v0.23.2.
Affected products
- Hugging Face tokenizers 0.23.1
Timeline
- 2026-09-04: disclosed