How do we deploy and evaluate an AI application in a CI/CD pipeline?

Every prompt, model or data change goes through a quality gate.

Picture it first

Before a new recipe goes on a restaurant menu, the chef tastes it (quick check), then a few staff try it (bigger test), then it's offered only to a few tables, and finally to everyone. If customers don't like it, the old recipe comes back the same evening.

Now the same idea, in simple QA words

In AI projects, a change to a prompt, an AI model or the documents is as risky as a code change. So we test it automatically in the CI/CD pipeline. Every change runs quick checks first. Bigger tests run every night: the full set of test questions, safety attacks, cost and speed limits. If quality drops, the release is blocked. Releases go out slowly — first to a few users — and can be rolled back quickly.

How the story connects to the real thing

In the storyIn the AI system
Chef tasting the new recipeQuick checks on every change
A few staff trying itFull nightly test suite
Offering it to a few tablesCanary release to a small % of users
Old recipe back the same eveningQuick rollback

Real-world example

A team changes the system prompt of their claims bot. GitHub Actions runs: pytest + JSON schema checks (1 min), 30-case DeepEval smoke set (4 min) → PR passes. Nightly: 600-case golden set + 200 red-team prompts + cost/latency budget. Faithfulness drops from 0.91 to 0.84 on the 'exclusions' category → release blocked. The fix is merged, scores recover, and the change goes to 5% canary, then 100%.

Step by step, in plain words

  1. Save prompts, model names and test data in Git like code.
  2. On every change: run quick tests.
  3. Every night: run the full test set + safety tests + cost/speed checks.
  4. If any important score drops: block the release.
  5. Release to a few users first, then everyone.
  6. Keep the old version ready to roll back.

How a tester tests it

  • Make thresholds explicit and reviewed (e.g. faithfulness ≥ 0.85, safety pass = 100% on critical set).
  • Keep PR evals fast and cheap; push expensive evals to nightly.
  • Handle non-determinism: low temperature, repeated runs or tolerance bands to avoid flaky gates.
  • Test the rollback path before you need it.

Try it yourself

1) Write a pipeline outline (stages + what runs + pass criteria) for a RAG chatbot in your favourite CI tool. 2) Decide which checks run on every PR and which only nightly.

Interview angle

“How do you prevent flaky LLM evaluations from blocking your CI pipeline?”

I separate deterministic checks from scored evals, run the model at low temperature, use tolerance bands or average over repeated runs for borderline metrics, pin model and judge versions, and keep the PR gate small and focused. Flaky cases get investigated, not silently retried.