Tokenization: From Letter to Token

How does a model actually process language before it responds?

Tokenization: From Letter to Token

Or: Why AI "ChatGPT" sometimes reads as three words

Imagine you want to explain to an AI what a sentence is. Sounds trivial. It's not.

Before a language model even begins to ponder your question, it needs to break your text down into small units. These units are called tokens — and they're not what you might think.

Tokens are not words.

The word "Tokenization" might be two or three tokens for a model: "Token", "iz", "ation". The word "AI" is one token. "Ärger" (anger) — the 'ä' could trip up the model, depending on the training language. Numbers are often tokenized digit by digit: "1984" → "19", "84".

Why? Because models are not trained on whole words, but on statistical fragments. The tokenizer — its own algorithm, usually Byte Pair Encoding (BPE) — learns which character sequences occur particularly frequently and bundles them. This creates "words" for the AI: Not grammatically, but statistically.

This has consequences.

First: Cost. API prices are measured in tokens. A long letter of 800 words might be 1,100 tokens. Those who don't know this will be surprised by the bill.

Second: Linguistic inequality. English texts are tokenized more efficiently than German ones. German has compound words ("Donaudampfschifffahrtsgesellschaft") which are a challenge for tokenizers. Turkish, Arabic, Chinese? Even more complex. This means: Non-English users pay more — for the same conversation.

Third: Oddities. Tokenization explains why AI sometimes fails with words that are trivial to us. Proper nouns, neologisms, dialects, technical terms — they rarely appear in training, are therefore split into many small tokens, and the probability of a coherent prediction decreases.

The sociological perspective: Whose language is preferred?

Tokenization is not neutral. It is a decision about representation: Which languages were preferred in training? Which writing systems are "built-in" to the tokenizer?

Bourdieu would ask: Who has linguistic capital in this system? And Nancy Fraser would add: Which counter-publics — which languages, dialects, scripts — are systematically disadvantaged because their linguistic volume is underweighted in the tokenizer?

Tokenization is the infrastructure of inequality. Unobtrusive, technical, deep.

Try it now (5 minutes):

Go to platform.openai.com/tokenizer and enter the same sentence in English, German, and Turkish. Count the tokens. What do you notice?

Reflection Question:

If your native language is more expensive to tokenize than English — is that a technical detail or a political decision?

Diskutiere diesen Text, seine Begriffe oder Denker:innen mit Sociologica. Dialektisch (Lesart + Gegen-Lesart), mit Leitfrage zurück an dich.

Diskussion starten
Day 13