Skip to content

Create a golden evaluation dataset to regression-test prompt updates

Stop testing system prompt edits by typing three random questions by hand—run 150 real edge-case scenarios automatically on every CI change.

Copy the prompt and paste it into the AI assistant you already use.

Step-by-step golden eval dataset construction

  1. 01

    Sample real messy user phrasings, not clean prose

    Pull anonymized real transcripts—complete with typos, shorthand, and multi-part asks—rather than sanitized questions written by engineers.

  2. 02

    Cover four mandatory scenario buckets

    Include 50% core answerable intents, 20% multi-turn tool/routing flows, 15% out-of-KB 'must abstain' gaps, and 15% guardrail/injection probes.

  3. 03

    Define deterministic + rubric assertions per case

    Assert exact expected tool calls and router picks in code, and use a narrow binary rubric ('Mentions 14-day cap: Yes/No') for prose grading.

  4. 04

    Gate CI deployments on zero P0 regressions

    Block prompt or chunking merges automatically if safety/policy pass rates dip below 100% or grounded accuracy drops more than 2%.

How it works in practice

Eval test case #084 (Abstention + Commercial Guardrail)

Input: 'Can I pause my annual Enterprise contract for 3 months while our VP is on sabbatical?' | Ground truth KB: Pauses exist on Monthly Starter/Growth only; Annual Enterprise has no pause option.

Automated CI assertion rubric

• `router_target == 'billing_specialist'` [PASS] • `must_not_promise_pause == true` [PASS] • `cites_chunk == 'doc_annual_terms_s2'` [PASS] • `offers_csm_escalation == true` [PASS] Latency: 1.42s | Cost: $0.0031 | Verdict: PASS (4/4)

Binary per-case assertions turn subjective 'vibe checks' into repeatable engineering CI gates.

Questions and answers

How many test cases do we need in v1 of a golden chatbot dataset?

Start with 80 to 120 well-curated cases—that takes one focused afternoon to label and catches 85% of prompt and model regressions.

Why shouldn't we ask an LLM judge to grade replies on a vague 1-to-10 scale?

1-to-10 scales drift wildly across runs (scoring 7 vs 8 arbitrarily); narrow binary Yes/No checks per factual criterion stay deterministic.

How should our golden dataset evolve after launch?

Every time a real production chat hallucinates or fails QA, sanitize that exact input and append it as a permanent regression case.

Learn more

Related guides and background articles for this task