Lesson 9 of 28 · Language models and chat assistants

What is a token in AI?

A token is a small piece of text, such as a word, part of a word, a punctuation mark or a space, that a language model reads and writes. Models split everything into tokens before processing it and generate replies one token at a time. Tokens also measure usage: context limits and API prices are counted in tokens.

Why models use tokens instead of words

Computers work with numbers, so a model needs a fixed dictionary, called a vocabulary, that maps pieces of text to numbers. A vocabulary of whole words would be enormous and would still miss new words, names and typos. Spelling everything out letter by letter would make every sentence very long. Tokens are the compromise: common words get their own token, and rare words are built from smaller pieces.

Think of LEGO. Frequent words are ready-made bricks, while unusual words are assembled from several small bricks. A widely used way of building such vocabularies, byte-pair encoding, was applied to neural translation by Sennrich et al. in 2015 and is now common in language models.

How text turns into tokens

A tool called a tokenizer does the splitting before the model sees anything. As a rough guide, OpenAI's tokenizer page says one token is about four characters, or three-quarters of a word, of common English text. Other languages and other models differ, so use a tokenizer tool for real counts. Some common patterns:

Short, common English words such as "the" or "cat" are usually one token.
Long or rare words are split into pieces, for example "tokenization" into "token" and "ization".
Spaces, punctuation and numbers count too, and long numbers are often split into chunks.
Chinese depends heavily on the tokenizer: one character may be one token, several characters may share a token, or one character may take more than one.
Multimodal models turn images and audio into tokens as well, which is why a picture uses up part of the context window.

Why tokens matter to you

Tokens are the unit behind most of the numbers you see on AI product and pricing pages:

Cost: AI APIs charge per token, usually with separate rates for input (what you send) and output (what the model writes).
Limits: the context window, the amount a model can read at once, is measured in tokens, and so are maximum reply lengths.
Speed: models generate one token at a time, so longer answers take longer.
Language: the same message can use a different number of tokens in another language or another model.

What a token is not

A token is not a word and not a character, although it sometimes happens to be one. In AI it also has nothing to do with the word's other common meanings: a crypto token is a digital asset recorded on a blockchain, and an access token or API token is a password-like key that proves who you are to a service. On a language-model pricing page, "tokens" always means pieces of text.

Example

Splitting a sentence into tokens

Worked example

Text: "Tokenization isn't hard!" Possible tokens (illustrative): [Token] [ization] [ isn] ['t] [ hard] [!] → 6 tokens for 3 words Each token is then swapped for its ID number in the model's vocabulary. The model only ever sees those numbers.

Exact splits vary between models. Paste your own text into a tokenizer tool, such as OpenAI's, to see real token boundaries and counts.

Key points

A token is a chunk of text: a word, part of a word, punctuation or a space.
Models read and write tokens, not words or characters.
Context windows and API prices are measured in tokens.
The same text can produce different token counts in different models and languages.
In AI, "token" has nothing to do with crypto tokens or login tokens.

Try it on Mindber

Turn token counts into cost estimates

AI pricing explained

Glossary: token-based pricing

Next lesson: What is a context window?

Frequently asked questions

How many words is 1,000 tokens?

It depends on the language and the model's tokenizer. For common English text, OpenAI's rule of thumb is about three-quarters of a word per token, so 1,000 tokens is roughly 750 words. Chinese and other languages can differ considerably. For exact counts, use the provider's tokenizer or the usage figures in its API responses.

Why do AI providers charge per token?

Tokens track the work a model actually does: every input token has to be processed and every output token generated. Charging per token makes cost scale with use. Output tokens are usually priced higher than input tokens, because generating text takes more computation than reading it.

Is an AI token the same as a crypto token?

No. In language models, a token is a piece of text. In blockchain, a token is a digital asset recorded on a ledger, and in software security an access token is a credential that proves who you are. The word is the same, but the ideas are unrelated.