How does tokenization work inside ChatGPT or Claude?
Models don't read letters or words. They read numbered chunks.
Picture it first
Think of LEGO. Common shapes like 'the', 'and', 'cat' come as one ready-made brick. A weird word like 'Supercalifragilistic' doesn't have its own brick, so it's built from smaller bricks: 'Super' + 'cal' + 'ifrag' + 'ilistic'. The computer then gives each brick a number — like a sticker on the box — and only works with the numbers.
Now the same idea, in simple QA words
Computers can't read letters the way we do — they only understand numbers. So before the AI reads your text, it cuts the text into small pieces (like LEGO bricks) and gives each piece a number. These pieces are tokens. Common words are one piece. Long or rare words are cut into 2–4 pieces. Why QA should care: AI companies charge money per token, and every AI has a token limit. So more tokens = more cost and less space.
How the story connects to the real thing
Real-world example
Indian testers see this all the time: an English sentence of 10 words may be ~13 tokens, but the same sentence in Hindi or Tamil can be 2–4x more tokens, because those scripts are less common in the tokenizer's training data. Same meaning, higher cost and less room in the context window. Long IDs like 'INV-2024-000781' or JSON with lots of braces also eat tokens quickly.
Step by step, in plain words
- You type a sentence.
- The tokenizer cuts it into small pieces.
- Each piece is swapped for its ID number.
- The AI works only with these numbers.
- The answer numbers are turned back into words for you.
How a tester tests it
- Use the provider's token counter (e.g. tiktoken for OpenAI, the count-tokens API for Claude) to build cost estimates per test case.
- Test multilingual inputs: same question in English, Hindi, emoji-heavy text — compare token counts and truncation.
- Test the input-length limit with real token counts, not character counts — a 'max 4,000 characters' UI rule is not a token rule.
- Watch for 'glitch' inputs: very long numbers, repeated characters, code — they tokenize strangely and can cause odd output.
Try it yourself
1) Paste a paragraph into an online tokenizer playground (e.g. OpenAI's Tokenizer page) and count tokens vs words. 2) Translate the same paragraph into your native language and count again.
Interview angle
“Why can't an LLM reliably count the letters in 'strawberry'?”
Because it never sees the letters. It sees tokens like 'str' + 'awberry' as ID numbers. Counting characters inside a token requires knowledge it only has indirectly. It's a tokenization limitation, which is why character-level tasks should be tested explicitly and often handed to code tools instead.