Eval Engineering Playbooks
Playbooks · Eval engineering overview · Golden datasets →
Implementation guides for the Eval Blueprint. G.A.I.N Evaluation covers principles. These playbooks are the how: data sources, scorers, and plane-specific gates.
THE CLAIM
Eval is a control system, not a scoreboard. Build golden cases first, score every plane that can fail, and promote production signals back into the dataset. One end-of-pipeline number is not enough.
Two playbook groups
| Group | Overview | What you build |
|---|---|---|
| Methods | Golden datasets → | Datasets, synthetic coverage, online/dynamic scoring, human review, LLM-as-judge |
| Planes | Input → | Per-plane gates from Input ① through Outcome ⑧ |
Plus Further reading (external) for third-party eval tools and guides mapped to this series.
Recommended path
- Golden datasets: versioned, plane-tagged cases before automation
- Synthetic generation and Human review: expand coverage and calibrate rubrics
- LLM-as-judge: scale scoring once humans set ground truth
- Plane playbooks: gate each plane your system actually owns (start with failure modes you already see)
- Online & dynamic: score live traffic and feed incidents back into goldens
Bridge reading: Eval Blueprint · G.A.I.N Evaluation.
Method playbooks at a glance
| Playbook | One-line purpose |
|---|---|
| Golden datasets | Versioned, plane-tagged, risk-tiered case libraries |
| Synthetic generation | Edge and adversarial coverage without polluting goldens |
| Online & dynamic | Live sampling, shadow scoring, drift, promote-back loops |
| Human review | Sampling, rubrics, adjudication that anchors automation |
| LLM-as-judge | Scaled judging with bias controls and human calibration |
Plane playbooks at a glance
| # | Playbook | One-line purpose |
|---|---|---|
| ① | Input | Parsing, intent, injection resistance, PII before inference |
| ② | Data | Source freshness, lineage, access boundaries |
| ③ | Context | Retrieval precision, ranking, packing, abstention |
| ④ | Reasoning | Faithfulness, conclusion quality, multi-step logic |
| ⑤ | Tool | Selection, arguments, idempotency, schema compliance |
| ⑥ | Memory | Session scope, TTL, cross-session leakage |
| ⑦ | Action | Policy enforcement, authorization, audit before side effects |
| ⑧ | Outcome | Task success, clarity, usefulness, trust in the final response |
Who should read what
| Role | Start with | Then |
|---|---|---|
| AI platform | Golden datasets, LLM-as-judge | Online & dynamic, relevant plane playbooks |
| Domain squad | Golden datasets, Human review | Plane playbooks for your surface (e.g. Context, Tool) |
| Governance / risk | Human review, Action | Input, Outcome |
| RAG / knowledge | Data, Context | Eval Blueprint |
| Agent / PGAR teams | Tool, Action | PGAR Runtime, Memory |