How can we reduce token consumption and API costs?
Every token is a coin. Stop throwing coins away.
Picture it first
Imagine sending text messages that cost 1 rupee per word. You'd stop writing 'Hello my dearest friend, I hope you are doing wonderfully well today' and just write 'Hi! Coming at 5?'. You'd also ask your little brother (cheaper) for easy answers and only call the expensive tutor for hard maths homework.
Now the same idea, in simple QA words
You pay for every token you send and every token the AI writes back. So to save money: send less text, ask for shorter answers, use a cheaper small AI for easy jobs, and re-use work instead of repeating it. One rule for QA: after every cost-saving change, re-test quality — cheaper is only good if answers are still correct.
How the story connects to the real thing
Real-world example
A QA team's test-case generator sends a 6,000-token system prompt + the whole Jira epic on every call, using the biggest model. After optimisation: the system prompt is cached, only the relevant user story is sent, a small model drafts test cases and a large model only reviews tricky ones, and output is capped as structured JSON. Cost per run drops by well over half while quality scores stay flat on the golden set.
Step by step, in plain words
- Measure: how many tokens and how much money each request uses.
- Remove unnecessary text from prompts.
- Send easy tasks to a cheaper model.
- Ask for shorter, structured answers.
- Re-use repeated parts (caching).
- Re-test quality to make sure nothing broke.
How a tester tests it
- Every cost optimisation is a change — re-run the golden dataset and compare quality before/after.
- Add cost-per-request and tokens-per-request as tracked metrics with alert thresholds.
- Test routing logic: are hard questions ever wrongly sent to the small model?
- Test cached answers for staleness: does a cached response survive a policy update?
Try it yourself
1) Write a prompt, count its tokens, then rewrite it 40% shorter and compare the outputs. 2) Ask for the same answer 'in under 50 words' and compare output token counts.
Interview angle
“Your AI feature's monthly bill doubled. How do you find the cause?”
I break cost down by endpoint, model and prompt version using token logs. Typical causes: a prompt change that added examples, retrieval returning more chunks, a new feature using the big model by default, agent loops calling the LLM repeatedly, or a cache-hit rate drop because someone put a timestamp at the start of the prompt. Then I fix it and prove quality didn't regress.