30 docs tagged with "ai-intelligence"

Hub for Agents playbooks: intent routing, orchestration (session, memory, autonomy), and tool manifests.

Run Pattern 0-3 from the pinned route: single inference, autonomous loop, fixed workflow, or guided hybrid inside Plane ②.

Eval-gated model swap and rollback at the LLM gateway: change records, traffic split, and when canary becomes stable.

Versioned registry of approved model endpoints per profile, task, data class, and region: the Plane ③ source of truth.

Hub for eval engineering playbooks — golden datasets, synthetic generation, online scoring, human review, LLM-as-judge, and per-plane recipes.

How to evaluate the Input plane — parsing, intent, injection resistance, and PII handling before inference begins.

How to evaluate the Data plane — source freshness, lineage, access boundaries, and factual correctness of underlying knowledge.

How to evaluate the Context plane — retrieval precision, ranking, scope, packing, and abstention when evidence is thin.

How to evaluate the Reasoning plane — faithfulness to context, conclusion quality, tool selection, and multi-step logic.

How to evaluate the Tool plane — selection, arguments, idempotency, error handling, and schema compliance for agent tool calls.

How to evaluate the Memory plane — session scope, TTL, consistency, and cross-session leakage in agent and copilot systems.

How to evaluate the Action plane — policy enforcement, authorization, side effects, and auditability before irreversible operations execute.

How to evaluate the Outcome plane — end-user task success, clarity, usefulness, and trust in the final delivered response.

Curated third-party articles, guides, and tool docs on LLM and agent evaluation — mapped to the Eval Framework Blueprint series.

Per-call filter, score, and pick (or abstain) at the LLM gateway: task type, failover, cost caps, and residency.

How to design, version, and maintain golden datasets for plane-aware evaluation — representative tasks, edge cases, adversarial cases, and production replays.

Playbooks for human evaluation in production AI — sampling strategy, rubrics, adjudication, and how manual scores anchor automated and LLM-as-judge gates.

Every plan or synthesize call goes through Plane ③, then validate proposals against the manifest and gate side effects at the PEP.

Hub for intent routing playbooks: route contract reference, route table lifecycle, layered classification, agentic app wiring, and routing eval CI.

Implement intent classification: eligible routes, rules (channel and event), classifier, LLM fallback, safety veto, and outcomes before the agent loop.

How to deploy LLM-as-judge for plane-aware evaluation — rubric design, judge selection, bias controls, and calibration against human ground truth.

Hub for Plane ③ model routing playbooks: capability matrix, gateway task routing, and eval-gated canary promotion at the LLM gateway.

How to run online evaluation on live traffic — sampling, shadow scoring, canary eval, drift detection, and promoting production signals back into golden datasets.

Hub for Plane ② orchestration playbooks: session custody, memory, autonomy shape, and inference handoff after the intent router decides and the app pins the run.

Field dictionary for route table rows: route, activation target (which agentic app to start), manifest, policy, retrieval, memory, workflow, and related artifact ids.

Where route contracts live, how to version and promote them, entitlement requirements per row, and rollback in regulated environments.

Golden intent sets, release gates, adversarial coverage, and incident replay for Plane ① routing — aligned with Eval Input plane.

Two pins, durable run store, and when a checkpointer or Temporal is enough: loop bounds, credential stripping, restart by run_id.

How to generate synthetic eval cases for edge and adversarial coverage — without polluting golden datasets or optimizing for the generator.

Decide-only router: freeze the route, async-start the app, skip the loop on clarify/abstain. UI and event ingress.