How does an LLM generate an answer, token by token?

It doesn't write sentences. It predicts the next piece, again and again.

Picture it first

Imagine playing a game where you have to finish the sentence 'Once upon a...'. You'd shout 'time!' because that's what usually comes next. Now you say the whole thing again — 'Once upon a time' — and guess the next word: 'there'. Then 'was'. An LLM plays this guessing game super fast, one tiny piece at a time, until the story is done. It never sees the end of the sentence before it gets there.

Now the same idea, in simple QA words

Remember the sentence-finishing game? That is exactly what ChatGPT or Claude does. It does not think of the full answer first. It looks at your question, guesses the next small piece of the answer, writes it, then guesses the next piece, and so on. Each small piece is called a token. This is why you see the answer appear slowly on the screen, word by word. And because it is guessing, the same question can give a slightly different answer each time — that is normal, not a bug.

How the story connects to the real thing

In the storyIn the AI system
Finishing 'Once upon a…'Reading your question
Shouting the next word: 'time!'Predicting the next token
Saying the whole sentence again and adding one wordAdding the new token and predicting again
Stopping when the story is finishedStopping at an 'end' signal or a word limit

Real-world example

When you ask ChatGPT or Claude 'Write a leave application', you see the words appear gradually on screen. That's not an animation effect — it's streaming: each token is sent to you the moment it is chosen. If the model chose 'Dear' as token one, everything after is shaped by that choice. That's also why a long answer costs more and takes longer: every single token is one more full trip through the model.

Step by step, in plain words

  1. You type a question.
  2. The AI guesses the most likely next word-piece.
  3. It adds that piece to the answer.
  4. It reads everything again and guesses the next piece.
  5. It repeats until the answer is complete (or the length limit is reached).

How a tester tests it

  • Run the same prompt 5–10 times and record variation — that's sampling at work, not a bug.
  • Check behaviour at the max_tokens boundary: is the answer cut mid-sentence? Does the app detect finish_reason = 'length'?
  • Measure time-to-first-token and tokens-per-second separately — users feel them differently.
  • Verify stop sequences actually stop output (e.g. the bot must not continue writing the user's next turn).

Try it yourself

1) Ask a chatbot the same question 3 times with 'regenerate' and compare the first 10 words of each answer. 2) Ask it to 'answer in exactly 5 words' and see if the token-by-token nature makes it miscount.

Interview angle

“Why does the same prompt give different answers on different runs?”

Because the model samples the next token from a probability distribution. With temperature above zero, lower-probability tokens sometimes get picked, and one different early token changes everything after it. Even at temperature 0, GPU arithmetic and batching can cause tiny differences, so I test with properties and thresholds, not exact strings.