Token

Before a model processes text, a tokenizer converts it into a sequence of tokens from a fixed vocabulary, often using subword methods such as byte-pair encoding. Common words may be a single token, while rare words are split into several pieces. The model then works entirely with these token IDs and produces its answer token by token.

Tokens matter because they drive cost and capacity. API providers charge per token for input and output, and the context window caps how many tokens a model can consider at once. The same text can produce different token counts in different models, and in some tokenizers non-English text, including Chinese, takes more tokens to express the same meaning, which affects cost.

Example: a long word such as "internationalisation" may be split into several tokens, while "the" is usually one. Most providers offer a tokenizer tool or report exact token counts in API responses.

Limits: token counts are model-specific, so estimates based on word counts are approximate. Use the provider's usage reports for real figures.