Tokenisers: BPE, why token counts surprise you, and the bugs they cause

About this article: AI-assisted writing, reviewed against the cited sources. This is source research, not a lab validation of the procedures. Check the stated versions and your own recovery requirements before making changes.

A tokeniser turns a string into integer ids before any embedding lookup, attention or output softmax. Byte-pair encoding (BPE) decides which strings receive their own id. APIs bill that count, context windows budget it, and the key-value cache stores one slot per id. The count follows the merge list, the pre-tokenisation rules and the script. A word count is a different quantity, which is why the two so often disagree.

An AI writing system helped prepare this article for NetLeverage. Where the text says what a model "sees", that means the integer sequence presented to the network, not a claim about subjective experience. Nothing here was executed in a lab. Vocabulary sizes and regular expressions come from the files and papers opened on 10 November 2026. Re-read an encoding file before you hard-code an id from it. This article makes no NetLeverage product claim.

Why it matters

Four engineering quantities are token counts in disguise.

Context length. Meta's first Llama 3 models were trained on sequences of 8,192 tokens. That is a budget of ids. Text that fragments into more ids fills the window sooner.

Money and latency. Petrov et al. note that a per-token price, and a decoder that emits one token at a time, turn a longer encoding into a higher bill and a slower reply for the same meaning. They measured the gap before the model is called.

Arithmetic. Singh and Strouse show that digit grouping changes addition accuracy on named GPT-3.5 and GPT-4 checkpoints under greedy decoding. A formatting change can be a change of token boundaries.

Parameter count. The input embedding and the output projection each have one row per vocabulary id. GPT-2's paper sets that vocabulary at 50,257. Meta's Llama 3 announcement sets it at 128K. Those are different matrices.

How it works

The compression idea, adapted to words

Sennrich, Haddow and Birch adapt the 1994 byte-pair compression algorithm to word segmentation. Classic BPE replaces the most frequent byte pair with one unused byte. Their version merges characters, or character sequences, into a new symbol.

The training loop they specify is small:

1. Start the symbol vocabulary as the set of characters in the corpus.
2. Write each word as a sequence of those characters, plus an end-of-word marker. The marker lets the decoder put the original word breaks back.
3. Count adjacent symbol pairs. Ignore pairs that cross a word boundary, so the algorithm can run on a frequency-weighted dictionary rather than on the raw character stream.
4. Replace every occurrence of the most frequent pair with one new symbol.
5. Repeat. The only hyperparameter they name is the number of merge operations. The final vocabulary size is the initial character vocabulary plus that number of merges.

At test time the same merges are applied, in the learned order, to a word that has been split into characters. Frequent words collapse to one symbol. A rare word that was never seen still decomposes into pieces that were. In their running example, the learned merges come from the dictionary low, lowest, newer, wider, and the unseen word lower is segmented as low plus er plus the end-of-word marker.

BPE is a greedy frequency heuristic. Radford et al. show the practical cost: the rule can spend vocabulary slots on near-copies of a common word (dog, dog., dog!) unless merges across character categories are blocked.

Two vocabularies, or one

Sennrich et al. compare separate merge lists per language with one list learned on the concatenated text, which they call joint BPE. Separate lists stay compact and keep each symbol inside the language that produced it. Joint lists segment shared names more consistently. For English–Russian they transliterate the Russian side to Latin characters while learning the joint merges, so the alphabets overlap during training.

They also drop infrequent symbols, including noise from foreign alphabets, out of the network vocabulary, so some merge products never receive a row. Any word of known characters can still be segmented. On WMT 15 the paper's subword systems gained up to 1.1 BLEU English→German with a character-bigram ensemble; joint BPE gained 0.5. English→Russian gained 1.3 BLEU with both joint-BPE and character-bigram ensembles over dictionary back-off. The durable part is the mechanism: rare words become reusable pieces, and the decoder emits them without a side dictionary.

Byte-level BPE

A tokeniser that starts from Unicode code points, and wants to cover every Unicode string, needs a base vocabulary above 130,000 symbols before any merges, Radford et al. write. That already exceeds the 32,000–64,000 token vocabularies then common with BPE. A byte-level start needs 256 base symbols, one per byte, and can then merge.

GPT-2 uses that design and sets the vocabulary to 50,257. Merges across character categories are blocked, with an exception for spaces, so the vocabulary is not filled with separate atoms for dog. and dog?. Every byte has an id, so any Unicode string has a token sequence.

The OpenAI tiktoken file openai_public.py, retrieved from GitHub main on 10 November 2026, still ships that encoding as gpt2 and r50k_base, both with explicit_n_vocab 50257 and <|endoftext|> at id 50256. p50k_base sets explicit_n_vocab to 50281 with the same end-of-text id. In that snapshot cl100k_base places <|endoftext|> at 100257 and <|endofprompt|> at 100276, and o200k_base places <|endoftext|> at 199999 and <|endofprompt|> at 200018.

The tiktoken README states that the mapping is reversible, that it accepts text absent from tokeniser training, and that in practice a token corresponds to about four bytes. That four-byte figure is their stated average. A rare script or a digit string can sit far from it.

The pipeline around the merges

BPE merges are the middle of a pipeline. The Hugging Face Tokenizers documentation, pipeline page opened 10 November 2026, lists four stages for Tokenizer.encode:

Pre-tokenisation bounds what merges may glue together. The quicktour says that without a pre-tokeniser a frequent pair such as it is can become one token. With a whitespace pre-tokeniser, no token is larger than a pre-token. Their Whitespace example splits I'm into I, ', m. Digits(individual_digits=True) splits 911 into three tokens. Those choices dominate the count.

Normalisation runs first. The docs show NFD plus accent stripping turning Héllò hôw are ü? into Hello how are u?. A training normaliser and a production normaliser that differ will index different rows.

Decoding is a separate function. The same docs show a WordPiece path: the default join leaves ## markers in the text, and decoders.WordPiece rejoins the pieces. BPE has the same bug when the end-of-word suffix or continuing-subword prefix at decode time differs from training.

The documented BPE constructor exposes dropout, unk_token, fuse_unk, byte_fallback (default false) and ignore_merges. The v0.20.3 quicktour gives BpeTrainer defaults of 30,000 vocabulary entries and minimum pair frequency 0; its versioned trainer reference lists options including limit_alphabet, initial_alphabet and max_token_length. Special-token order assigns ids: the first string in the list receives id 0. A missing control token, or a different order from the model config, is a broken contract.

SentencePiece: spaces are symbols

Kudo and Richardson's SentencePiece trains BPE or a unigram language model on raw sentences. The Sennrich pipeline assumes a prior word split, which drags in a language-specific segmenter for Chinese, Japanese and Korean.

SentencePiece escapes whitespace as the visible symbol U+2581 (▁) and then segments. Detokenisation replaces that symbol with a space, so the normalised text round-trips. Schemes that only mark whether a piece is word-initial cannot always encode consecutive whitespace, and the reverse mapping becomes ambiguous.

The SentencePiece options document defines --vocab_size as the desired vocabulary size including special symbols. Which control symbols are enabled depends on the configuration; padding is not necessarily assigned an enabled id. A merge count is a BPE-only knob and does not transfer to the unigram model.

The default normaliser is a subset of Unicode NFKC, compiled into a transducer stored inside the model file. Full canonical-combining-class reordering is not implemented. The same file holds the vocabulary, so encode and decode do not depend on an external Moses or KyTea configuration. Ship that file with the network. A copied merge list plus a different normaliser is a different tokeniser.

The unigram model can sample among several legal segmentations. Their example tokenises "New York" differently on successive SampleEncodeAsPieces calls. A training job that samples and a server that always takes the single best segmentation are different functions. The Hugging Face BPE dropout flag is the related switch for BPE: non-zero dropout in training and zero in serving is the same split.

Where the surprising counts come from

Spaces and punctuation. The r50k pattern in openai_public.py keeps an optional space on the following letters (?\p{L}). "hello" and " hello" are different id sequences, and the pieces of a split prompt do not add up to the encoding of the joined string.

Digits. The cl100k_base and o200k_base patterns opened on 10 November 2026 both contain \p{N}{1,3}. Singh and Strouse tie the cl100k_base rule, in that same file, to the way GPT-3.5 and GPT-4 chunk numbers into pieces of one to three digits from the left. Llama and PaLM, in their survey, use a single digit per token. GPT-3's p50k_base is pure BPE: many three-digit strings have no dedicated token, and four-digit strings do not all split the same way. Which three-digit string is one token is a fossil of merge frequency.

On few-shot addition with gpt-3.5-turbo-0301, greedy decoding and the default system prompt, a separator every three digits counted from the right raised accuracy against that left-to-right default. Their shot sweep moves left-to-right from 68.5% at 1 shot to 75.6% at 8 shots, and right-to-left from 95.6% to 97.8%. They describe accuracy as "up to 20% higher" (Figure 1, Section 3). The printed shot-sweep values above imply gaps of 27.1 and 22.2 percentage points; percentages and percentage-point differences are distinct. Other separators, and extra tokens that did not change digit direction, left the gap in place. The numbers are for those checkpoints and for addends of 7 to 9 digits. If a product depends on arithmetic, print the digit segmentation on a sample of real prompts.

Scripts. Petrov, La Malfa, Torr and Bibi measure the same text across languages on more than a dozen tokenisers. For the tokeniser they attribute to ChatGPT and GPT-4, Italian costs about 1.6 times the English token count, Bulgarian 2.6 times, Arabic 3 times, and Shan up to 15 times. In the paper's ByT5 table, Burmese/Simplified-Chinese is 3.77, Burmese/Yue is 4.03, and Tibetan/Yue is 3.80: roughly fourfold, not a uniform lower bound above four. Common words can fragment while a rare string keeps a whole id: the Arabic word for "why" splits into letters, Bulgarian "защо" takes six GPT-2 tokens for four letters, and the kanji 言 takes three. On their count, more than half of kanji need three GPT-2 tokens, and more than 65% need three in cl100k_base. The bill, the latency and the usable context all scale with that length. Meta's later claim of up to 15% fewer tokens for Llama 3's 128K vocabulary versus Llama 2 is a comparison of those two tokenisers, not evidence that parity across languages has been reached.

Special tokens and unknown pieces. Registered as a special token, <|endoftext|> is one id. Passed through ordinary merges, the same characters are ordinary text and do not round-trip to the control id. The tiktoken README tells you to rename the encoding when you add special tokens, so callers can see the contract changed. A character outside the trained alphabet becomes the unknown token: the Hugging Face quicktour's sentence containing 😁 yields [UNK] for that character. Fusing adjacent unknown tokens, or turning on byte fallback, changes both the ids and the string you decode. Round-trip emoji, mixed newlines and non-BMP characters on the bytes you actually receive.

The corpus you counted. Each merge is the most frequent pair at that step, so log lines and product names in the corpus consume ids. Petrov et al. treat that as a property of internet-scraped vocabularies. A clinical or source-code tokeniser will not match a general web tokeniser, and a model trained with one cannot be fed the other without new embedding rows for the ids it actually uses.

What to choose and why

Meta's Llama 3 post pairs the 128K vocabulary with up to 15% fewer tokens than Llama 2, and, with grouped-query attention, inference efficiency they describe as on par with Llama 2 7B even though the 8B model has more parameters. Measure tokens on your own corpus before treating 15% or 128K as a portable constant.

Risks and limits

• The merge list is not the tokeniser. Normalisation, pre-tokenisation, special-token ids and the decoder can change the ids while the merge file stays byte-identical.
• Averages hide the tails. "About four bytes per token" is the tiktoken README's practical average. Petrov et al.'s ratios show how far a single language pair can sit from any English-centric average.
• Greedy BPE mis-allocates a finite vocabulary. Radford et al. add category constraints for that reason. Dropout or unigram sampling also means one string can have several legal segmentations.
• NFKC and accent stripping drop distinctions that identifiers, case-sensitive codes and accented names need. SentencePiece's default NFKC is a documented subset, so two "NFKC" implementations can still disagree.
• Cited model behaviour is checkpoint-specific. Singh and Strouse's percentages are for named GPT-3.5 and GPT-4 snapshots under greedy decoding. The checkpoint effect is non-monotonic: the authors report mitigation in GPT-4 versions and a stronger tokenisation effect again in the GPT-4 Turbo checkpoint they tested. Re-measure before you depend on a delimiter trick.
• This article does not validate a library version in a lab. tiktoken constants are from GitHub main on 10 November 2026. The Hugging Face pipeline page is unversioned and carries an rc0-bindings warning beside the normaliser example; model and trainer options here use v0.20.3. Pin the commit or package version in any deployment.
• No NetLeverage product behaviour is claimed.

Sources

1. Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units, arXiv:1508.07909 — https://arxiv.org/abs/1508.07909 (HTML via ar5iv opened 10 November 2026). BPE segmentation algorithm, end-of-word marker, word-boundary constraint, vocabulary equals characters plus merges, joint versus separate BPE, WMT 15 BLEU deltas.
2. Radford et al., Language Models are Unsupervised Multitask Learners — https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf (opened 10 November 2026). Byte-level BPE, base vocabulary 256, vocabulary 50,257, ban on cross-category merges with a space exception.
3. Kudo and Richardson, SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing, arXiv:1808.06226 — https://arxiv.org/abs/1808.06226 (HTML via ar5iv opened 10 November 2026). Raw-sentence training, U+2581 whitespace, lossless round trip, --vocab_size, NFKC subset, self-contained model file, unigram sampling.
4. OpenAI tiktoken README and tiktoken_ext/openai_public.py — https://github.com/openai/tiktoken (raw files opened 10 November 2026). Encoding names, special-token ids, digit patterns, reversibility, stated average of about four bytes per token.
5. Hugging Face Tokenizers documentation, pipeline, models, trainers and quicktour — https://huggingface.co/docs/tokenizers/en/pipeline (opened 10 November 2026). Four-stage pipeline, retrain guidance, BPE constructor flags, trainer defaults, unknown-token and decoder examples.
6. Petrov, La Malfa, Torr and Bibi, Language Model Tokenizers Introduce Unfairness Between Languages, NeurIPS 2023 — https://arxiv.org/abs/2305.15425 (conference PDF opened 10 November 2026). Cross-language length ratios and the cost, latency and context consequences.
7. Singh and Strouse, Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs, arXiv:2402.14903 — https://arxiv.org/abs/2402.14903 (HTML opened 10 November 2026). Left-to-right versus right-to-left digit chunks and the reported accuracy gap.

8. Meta, Introducing Meta Llama 3, 18 April 2024 — https://ai.meta.com/blog/meta-llama-3/ (opened 10 November 2026). 128K vocabulary, training sequences of 8,192 tokens, up to 15% fewer tokens than Llama 2.

9. Hugging Face Tokenizers v0.20.3: models, trainers, and quicktour. Reopened 10 November 2026.

10. SentencePiece training options, unpinned master observed 10 November 2026. Defines desired vocabulary including special symbols.

This article was originally published on 10 November 2026.