How do RAGAS and DeepEval measure AI response quality?

Under the hood: claims, questions, and judge prompts.

Picture it first

To check if your book report is honest, a checker breaks it into small sentences: 'The hero is a dog', 'He lives in Delhi', 'He wins a race'. Then they look in the book for each sentence. If 2 of 3 are in the book, your honesty score is 2/3. That's exactly how faithfulness works.

Now the same idea, in simple QA words

RAGAS and DeepEval are free tools that score AI answers automatically. Most scores work like a careful teacher. Example — Faithfulness (is the answer based on the documents?): the answer is broken into small facts, each fact is checked against the documents, and the score = facts supported ÷ total facts. Other scores check if the answer is relevant to the question and if the search found the right documents. DeepEval can also fail a test automatically when a score is below your limit.

How the story connects to the real thing

In the storyIn the AI system
Checker breaking your book report into small sentencesSplitting the answer into facts (claims)
Looking for each sentence in the bookChecking each fact against the documents
2 out of 3 found2/3 → Faithfulness score = supported ÷ total
Pass mark for the reportThreshold that passes/fails the test

Real-world example

Answer: 'The warranty is 2 years and covers water damage.' Context says: '2-year warranty; water damage not covered.' Claims: (1) warranty is 2 years → supported; (2) covers water damage → contradicted. Faithfulness = 1/2 = 0.5. With a DeepEval threshold of 0.8, this test fails in CI.

Step by step, in plain words

  1. Collect: question, AI answer, documents used (and correct answer).
  2. Break the answer into small facts.
  3. Check each fact against the documents.
  4. Calculate the scores.
  5. Compare with your pass mark (e.g. 0.8).
  6. Pass or fail the test.

How a tester tests it

  • Remember the metrics themselves use an LLM — pin the judge model and expect some variance.
  • Spot-check low and high scores manually; metrics can be fooled (e.g. empty context scoring high faithfulness, as Part 1 warned).
  • Choose metrics per stage: retrieval metrics for retriever changes, generation metrics for prompt changes.
  • Track metric cost — LLM-based metrics make many calls per sample.

Try it yourself

1) Take one chatbot answer and its source paragraph; manually extract claims and compute a faithfulness score. 2) Run the same sample through DeepEval's FaithfulnessMetric and compare.

Interview angle

“How is RAGAS faithfulness actually calculated?”

An LLM decomposes the answer into atomic claims, then checks whether each claim can be inferred from the retrieved context. The score is supported claims divided by total claims. So it measures grounding, not truth — if the context itself is wrong, a faithful answer is still wrong.