How Tokenization Works

Language models do not read letters or words. They read tokens, and the way text is split into tokens affects both cost and limits.

What a token is

A token is a chunk of text from a fixed vocabulary. It might be a whole common word, part of a longer word, a number fragment, a punctuation mark or whitespace. Before a model sees your text, a tokenizer converts it into a sequence of these chunks, each mapped to an ID.

How text is split

Many modern tokenizers use subword methods, such as byte-pair encoding. The vocabulary is built from patterns that appear often in training text. Frequent words tend to be one token, while rare words, names and technical terms are split into several pieces. For instance, an unusual word may be broken into a few fragments that each exist in the vocabulary.

Why counts differ between models

Each tokenizer has its own vocabulary and rules, so the same text can produce different token counts in different models, even from the same provider. An estimate that suits one model may be off for another.

What tends to use more tokens

• Source code, especially with unusual symbols or long identifiers.
• Numbers and random-looking strings such as IDs.
• Many non-English languages, which are often split into more pieces than common English text.
• Heavy formatting, such as repeated whitespace or large JSON structures.

Why it matters

Models have a context window, a maximum number of tokens they can handle at once, and providers usually bill per token. Knowing your token count helps you stay within limits and predict cost.

Rules of thumb and exact counts

For ordinary English prose, a common rough guide is about four characters, or three quarters of a word, per token. It is only a guide. The token counter uses these rules and labels its result as an estimate. For exact numbers, use your provider's own tokenizer or token-counting feature, or the usage figures returned with API responses.

Next, see How to Estimate LLM API Costs.