Canary Promotion
Blueprint · ← Gateway task routing · Canary promotion
A new model revision does not become production by updating a URL. It enters the capability matrix as canary, passes eval, then earns stable.
Swap the endpoint the same way you promote a route table: version, eval, pin, rollback. Console-only model changes are not a release process.
Promotion sequence
- Register the new endpoint in the catalog (revision, region, data class)
- Attach it as
canaryon the targetprofile_id+ task (bumpmatrix_version) - Run the offline golden suite for that profile and task
- Split a small share of live traffic to canary (same filters as stable)
- Compare quality, latency, cost, and abstain rate against stable
- On pass: repoint
stable, drop or keep canary for the next candidate - On fail: remove canary from the matrix; stable is unchanged
Offline gate (before live split)
Align scoring with Eval Reasoning plane for plan/synthesize quality. Task routing correctness is a gateway test, not a judge score.
| Gate | Bar |
|---|---|
| Golden task set | No regression vs current stable (baseline − 1% max) |
| Safety / abstain fixtures | 100% expected abstain or block |
| Schema / tool-proposal well-formedness | 100% on structured tasks |
| Cost estimate | At or under profile max_cost_usd_per_1k_tokens |
Attach eval_run_id to the change record that bumps matrix_version.
Online follow-up
| Signal | Action |
|---|---|
| Quality drop vs stable | Halt split; keep stable |
| Latency or error rate over SLO | Halt split |
| Cost per call over cap | Halt split even if quality is better |
| Abstain rate spike | Check filter; do not widen data class to recover volume |
| Shadow disagreement (canary ≠ stable) | Sample for human review before promote |
Canary sees the same data class, region, and entitlements as stable. A canary that only runs on public traffic is not evidence for confidential.
Rollback
| Event | Action |
|---|---|
| Canary failing | Delete canary pointer; no stable change |
Bad stable after promote | Repoint stable to prior endpoint id; bump matrix_version |
| Provider yank / CVE | Remove endpoint from catalog; matrix publish; in-flight pins keep old id until session ends |
In-flight sessions keep the matrix_version they pinned. New requests take active. Same rule as route table lifecycle.
Change record
Every matrix publish includes:
matrix_versioneval_run_id- Profile + task ids touched
stableandcanaryendpoint ids before and after- Approver (governance for model approval; platform for publish)
Failure classes
| Failure | Symptom |
|---|---|
| Swap model URL in app config | No eval, no rollback pointer |
| Canary on a weaker data class | False confidence at promote |
| Promote on latency alone | Quality regression in production |
No eval_run_id on the change | Cannot prove the gate ran |
Trace fields
matrix_version, model_route_id, canary, eval_run_id, stable_endpoint_id, canary_endpoint_id