Use case Q07ExecutionMedium

Failure analysis & flaky-test classifier

The problem

Every night 40 tests fail; someone spends the morning sorting real bugs from flaky noise.

Workflow

  1. CI resultsfailures + logs
  2. Groupby error signature
  3. LLM + historyreal bug / flaky / env / data
  4. Reportper group, with evidence
  5. Owneracts on 'real bug' first

Real-world example

38 failures collapse into 4 groups: 31 share 'ECONNREFUSED auth-service' (environment), 4 are known flaky timeouts, 2 assertion failures on discount price (real bug), 1 expired test user (data).

Starter prompt

Classify each failure group as REAL_BUG, FLAKY, ENVIRONMENT or TEST_DATA. Use the error, the last 10
runs history and recent commits. Give evidence for each. Return a table sorted by REAL_BUG first.

Tools

CI (GitHub Actions/Jenkins), test reports, LLM

Human check

Re-run 'flaky' tests before ignoring; a wrong 'flaky' label hides real bugs.

Measure success

Morning triage time; real bugs found before standup; flaky-test count trend.

Concepts behind it