Tokens — how AI reads | How AI Works

01 · Live tokenizer

The model never sees your words

It sees — chunks from a fixed . This is a real tokenizer (the o200k vocabulary) running in your browser. Every keystroke re-tokenizes.

Type1163␣anything6137␣—2733␣even1952␣emoji74471␣9552245🗼120␣or503␣code3490.13

38 characters → 11 tokens · 3.5 chars/token

Things to try

Click Numbers — "12345" splits unpredictably, which is one reason models are shaky at arithmetic. Click Whitespace — "hello" and " hello" (with a leading space) are different tokens entirely. Non-English text often costs 2–3× more tokens per word than English.

02 · Tokens are the billing unit

Every token is money

APIs charge per token — input and output separately. Paste any text and see what it costs at example rates.

Tokens

≈ Words

As input @ $3/M

$0.00000

As output @ $15/M

$0.00000

Example rates (typical mid-tier model pricing). Output costs ~5× input at most providers.

03 · Where the vocabulary comes from

BPE: merge the most frequent pair, repeat

The vocabulary isn't designed by hand. An algorithm reads a huge pile of text and repeatedly glues together the pair of symbols it sees most often. Step through it on a five-word corpus.

Step 0/7: start: characters

low

lower

lowest

newer

newest

After 7 merges the corpus needs just 8 symbols: low, new, er, est, e, r, s, t. Frequent stems and suffixes become single tokens; rare words stay decomposable.

The "strawberry" failure, explained

Ask a model how many r's are in strawberry and it often says 2. Tokenized, the word is something like [st][rawberry] — the model receives two IDs, never ten letters. Counting characters inside a token is like asking you how many brushstrokes are in a printed letter "A": the information was compressed away before you saw it.

Knowledge check

Did it stick?

1/3Why is "hello" a different token than " hello" (with a leading space)?

2/3A 1,000-token prompt costs $3 per million input tokens. Roughly what does it cost?

3/3Why do models miscount the r's in 'strawberry'?

Loading module…

Common questions this module answers

What is a token in a large language model?

A token is a chunk of text — usually a word piece of about 3–4 characters in English — that the model treats as a single unit. Models never see letters, only token IDs from a fixed vocabulary of around 100–200k entries learned with byte-pair encoding (BPE).

Why do AI models struggle to count letters in words like 'strawberry'?

Because the model sees the word as one or two tokens, not as individual letters. Counting the r's requires reasoning about characters inside a token it never sees spelled out.

How do tokens relate to API pricing?

LLM APIs price per token, for input and output separately. Roughly 4 characters of English is one token, so 1,000 tokens is about 750 words.