How tokens work in AI models

If you follow Token & Tech, you have already seen the word “token” in two different worlds. This article deals with only one of them: the token of language models, the unit of text that artificial intelligence works with.

An AI token is not a crypto token

It is worth separating the two right away.

In the crypto world, a token is a digital asset recorded on a blockchain. It can represent money, a right or a good.

In AI models, a token is a piece of text, such as a word, part of a word or a punctuation mark. It has no market value and cannot be bought or transferred. The two ideas only share the name.

From here on, whenever “token” appears, read it as a piece of text.

What a token is in a language model

A language model does not read letters the way you do. The repository of tiktoken, OpenAI’s tokenization library, puts it this way: language models “don’t see text like you and I”, but rather a sequence of numbers, the tokens.

The path is this:

1. the text you type goes through a tokenizer, a program that cuts it into pieces;
2. each piece corresponds to an identification number within a fixed list, the model’s vocabulary;
3. the model processes this sequence of numbers and returns new numbers;
4. the tokenizer converts the numbers back into readable text.

Google’s Gemini documentation describes the same principle: models process input and output at the granularity of tokens, which can be single characters or whole words, and long words are broken into several tokens. The process of slicing text is called tokenization.

Everything a model generates also comes out token by token. If you want to understand this generation process, see what LLMs are.

Why not use whole words or letters

It seems simpler to give each word a number. The Hugging Face documentation explains why that does not work well.

Word-level tokenization produces gigantic vocabularies, because each variation needs its own number: “love”, “loving”, “loved”, “lovingly”. Worse: any word outside the list becomes an “unknown” symbol, and the model has no idea what to do with it.

Character-level tokenization solves the problem of new words, since every text is made of letters. But the sequences become very long, and a single letter carries little meaning.

The solution adopted by most current models sits in between: subword tokenization. Common words become a single token. Rare words are assembled from known pieces.

The idea gained momentum with a 2015 paper by Rico Sennrich and co-authors, “Neural Machine Translation of Rare Words with Subword Units”. The authors started from an observation: translation is an open-vocabulary problem, and names, compounds and loanwords that no fixed dictionary anticipates keep appearing. By representing rare words as sequences of subwords, the system was able to handle them.

How the vocabulary is built

BPE, the most common method

The most widely used algorithm is BPE (byte pair encoding). According to Hugging Face, it is used in models such as Llama, Gemma and Qwen2.

How it works, simplified:

1. the vocabulary starts with individual characters only;
2. the algorithm goes through a large volume of text and finds the pair of neighboring symbols that appears most frequently;
3. that pair becomes a new symbol and enters the vocabulary;
4. the process repeats until the desired vocabulary size is reached.

An illustrative example, created by the editorial team: if “ç” and “ão” appear together very often in Portuguese texts, the pair “ção” tends to become a token of its own. Later, larger combinations, such as “ação”, may get theirs.

An important variant is byte-level BPE. Instead of starting from all possible characters, it starts from the 256 values of a byte. That way, any text can be represented without falling back on the “unknown” symbol. According to Hugging Face, GPT-2 uses this technique with a vocabulary of 50,257 tokens.

Other methods

The same documentation describes alternatives:

• WordPiece, used in the BERT family, which chooses the merges that most increase the likelihood of the training data;
• Unigram, used in models such as T5, which starts from a large vocabulary and gradually discards the least useful pieces;
• SentencePiece, a library that applies BPE or Unigram directly to raw text, treating the space as a symbol. This helps with languages that do not separate words with spaces, such as Chinese and Japanese.

A vocabulary is a design choice

The size and composition of the vocabulary change from one model to another. The Llama 3 technical paper reports a vocabulary of 128,000 tokens, combining 100,000 tokens from tiktoken with 28,000 additional tokens to better serve languages other than English.

The tiktoken README lists desirable properties of BPE: it is reversible and lossless (the original text can be reconstructed), it works on any text, even text outside the training data, and it compresses the text, since the token sequence is shorter than the byte sequence.

How much text fits in a token

There is no exact conversion, but there are references published by the makers themselves:

• the Gemini documentation, consulted in October 2026, states that a token is equivalent to about 4 characters and that 100 tokens correspond to about 60 to 80 English words;
• the tiktoken README says that, on average, each token corresponds to about 4 bytes;
• the Llama 3 paper reports that the new tokenizer improved compression, on a sample of English text, from 3.17 to 3.94 characters per token.

Doing the math with Gemini’s rule, a 1,000-word English text would come to between 1,250 and about 1,670 tokens. This is an estimate, not a measurement.

And Portuguese? We did not find an official reference with a specific ratio for the language. The rules above were described for English. To find the real number, count with the model’s own tool, as we show further below.

Same text, different counts

Each model has its own tokenizer, which is why the same text can produce different counts.

A concrete example appears in Anthropic’s token counting documentation: according to the page, consulted in October 2026, Claude models from version 4.7 onward use a new tokenizer that produces approximately 30% more tokens for the same text than previous models. The documentation itself recommends recounting prompts on the model you intend to use.

The difference also occurs between languages. A study presented at NeurIPS 2023 by Aleksandar Petrov and co-authors, “Language Model Tokenizers Introduce Unfairness Between Languages”, found cases in which the same text, translated into different languages, produced token sequences up to 15 times longer. According to the authors, this creates differences in cost, response time and how much content fits in the model, depending on the language.

Where tokens show up in practice

Context window

Every model has a limit on the number of tokens it can consider at once. That limit is the context window.

The Gemini documentation explains that the window is the combined limit of input and output tokens. Anthropic’s documentation on context windows compares the window to a “working memory” and details what counts toward it: system instructions, all the messages in the conversation, documents, images, tool definitions and the generated response itself.

The same page gives a useful warning: more context is not automatically better. As the number of tokens grows, accuracy and the ability to retrieve information tend to decline. That is why what you put in the window matters as much as the space available. The topic is covered in depth in context and memory in AI.

Cost

In pay-per-use services, the bill is usually calculated in tokens. The Gemini documentation states that the cost of a call depends, in part, on the number of input and output tokens. Prices change often and vary by model, so we do not list them here. Always check the service’s official price table, noting the date you checked.

Images, audio and video

Models that accept other formats also convert everything into tokens. According to the Gemini documentation, consulted in October 2026, an image of up to 384 by 384 pixels counts as 258 tokens, and audio consumes 32 tokens per second. These values are specific to that service and may change.

Usage limits

Many platforms limit how many tokens you can send per minute. Anthropic cites managing these limits and costs as one of the reasons to count tokens before sending a message.

How to count tokens before sending

You have three main options:

• tokenization libraries, such as tiktoken, which let you count tokens locally for compatible tokenizers;
• API counting functions, such as Gemini’s count_tokens, which returns the total input tokens before the call;
• the usage field in the response, which reports how many tokens were actually consumed.

One caveat: Anthropic warns that the advance count is an estimate and may differ slightly from the actual number. For planning, treat the result as an approximation.

Why this helps explain some errors

Because the model sees pieces of words, not letters, tasks that depend on individual characters can be harder for it. In the editorial team’s assessment, this helps explain why assistants sometimes stumble when counting the letters in a word or reversing a text character by character. It is not the only possible explanation, but it is a good starting point.

To write more efficient prompts, including saving tokens, see prompt engineering.

What to take from this article

In AI, a token is a piece of text that the model turns into a number in order to process it. It has nothing to do with crypto-assets.

Subword tokenization, with methods such as BPE, balances a compact vocabulary with the ability to handle new words.

Each model cuts text its own way. That is why rules like “4 characters per token” serve only as an estimate, especially outside English. When cost or context limits matter, count tokens with the tool of the model you are going to use.