66 docs tagged with "arch"

Prompt injection, PEP bypass, manifest violations, and entitlement escalation tests for PGAR runtimes.

Hub for Agents playbooks: intent routing, orchestration (session, memory, autonomy), and tool manifests.

Immutable verdict logs, examiner questions, and replaying authorization without chat transcripts.

Run Pattern 0-3 from the pinned route: single inference, autonomous loop, fixed workflow, or guided hybrid inside Plane ②.

Map business KPIs and customer journeys to observable events. Outcome SLOs, conversion signals, and business-layer instrumentation for unified observability.

Eval-gated model swap and rollback at the LLM gateway: change records, traffic split, and when canary becomes stable.

Versioned registry of approved model endpoints per profile, task, data class, and region: the Plane ③ source of truth.

Connect business KPIs to service traces and infrastructure signals. Correlation IDs, journey mapping, and the observability graph for unified enterprise operations.

Hub for eval engineering playbooks — golden datasets, synthetic generation, online scoring, human review, LLM-as-judge, and per-plane recipes.

How to evaluate the Input plane — parsing, intent, injection resistance, and PII handling before inference begins.

How to evaluate the Data plane — source freshness, lineage, access boundaries, and factual correctness of underlying knowledge.

How to evaluate the Context plane — retrieval precision, ranking, scope, packing, and abstention when evidence is thin.

How to evaluate the Reasoning plane — faithfulness to context, conclusion quality, tool selection, and multi-step logic.

How to evaluate the Tool plane — selection, arguments, idempotency, error handling, and schema compliance for agent tool calls.

How to evaluate the Memory plane — session scope, TTL, consistency, and cross-session leakage in agent and copilot systems.

How to evaluate the Action plane — policy enforcement, authorization, side effects, and auditability before irreversible operations execute.

How to evaluate the Outcome plane — end-user task success, clarity, usefulness, and trust in the final delivered response.

Curated third-party articles, guides, and tool docs on LLM and agent evaluation — mapped to the Eval Framework Blueprint series.

Curated third-party resources on PDP/PEP, OAuth, policy engines, and agent authorization, mapped to the PGAR playbook series.

Per-call filter, score, and pick (or abstain) at the LLM gateway: task type, failover, cost caps, and residency.

What examiners get before scale: inventory row, obligation map, runtime verdict sample, eval gates.

How to design, version, and maintain golden datasets for plane-aware evaluation — representative tasks, edge cases, adversarial cases, and production replays.

Two views, one G.A.I.N subject: estate operating how-to and PGAR runtime how-to.

Discipline rules for enterprise observability. Structured logs, correlation IDs, KPI mapping, alert ownership, and telemetry purpose in regulated environments.

Playbooks for human evaluation in production AI — sampling strategy, rubrics, adjudication, and how manual scores anchor automated and LLM-as-judge gates.

Every plan or synthesize call goes through Plane ③, then validate proposals against the manifest and gate side effects at the PEP.

Monitor compute, Kubernetes, network, and storage with signals linked to service identities. Infrastructure-layer observability for unified enterprise stacks.

Hub for intent routing playbooks: route contract reference, route table lifecycle, layered classification, agentic app wiring, and routing eval CI.

No inventory, no owner, no go-live. Minimum fields for an AI use-case row before runtime scale.

Implement intent classification: eligible routes, rules (channel and event), classifier, LLM fallback, safety veto, and outcomes before the agent loop.

How to deploy LLM-as-judge for plane-aware evaluation — rubric design, judge selection, bias controls, and calibration against human ground truth.

Where to maintain tool manifests, how agentic apps load them, versioning and rollback, and pros and cons of repo files vs registry APIs.

Tool manifests, schema compliance, PEP gating per tool, and blocking proposals outside the registry.

L0 through L5 observability maturity model for enterprises. Gap analysis, scoring checklist, and roadmap from reactive logs to business-aware and predictive operations.

Hub for G.A.I.N MCP: MCP is transport. Tool contracts (manifests) live under Agents.

Session, working, loop, episodic, and long-term memory as route policy plus a partitioned store: isolation, TTL, retrieve_only, writes, and erase.

Hub for Plane ③ model routing playbooks: capability matrix, gateway task routing, and eval-gated canary promotion at the LLM gateway.

Map each must-we obligation to a named control and an evidence artifact. Governance Operating how-to.

Hub for unified observability playbooks. Business, service, and infrastructure layers, correlation graph, governance, maturity, and operating model.

How to run online evaluation on live traffic — sampling, shadow scoring, canary eval, drift detection, and promoting production signals back into golden datasets.

RACI and team boundaries for unified observability. Product, engineering, platform, SRE, and architecture responsibilities in regulated enterprises.

Estate how-to for G.A.I.N Governance: inventory, obligation-to-control map, and go-live evidence pack. Runtime PEP/PDP lives in the Runtime series.

Hub for Plane ② orchestration playbooks: session custody, memory, autonomy shape, and inference handoff after the intent router decides and the app pins the run.

ALLOW, DENY, and STEP_UP only — policy versioning, rule authoring, and deterministic authorization.

The four steps every Policy Enforcement Point runs on every proposal: receive, ask PDP, audit, act.

API gateway, Identity Provider, token validation, and claims issuance at the trust boundary.

Session custody, orchestration, proposal routing, and receiving results before validation or synthesis.

Tool schemas only, proposal-not-permission, and keeping authority out of the model boundary.

The policy layer — enforcement point, decision point, verdict handling, and deny-before-downstream.

Re-authorization, side-effect execution, and returning results to the agentic app, not the LLM directly.

The five PGAR trust boundaries in request order (ingress, agentic app, LLM proposal, PEP + PDP, downstream), including multi-agent workflows, with links to each implementation playbook.

Core PGAR building blocks in implementation order — SARAC contracts, token custody, PEP/PDP enforcement, step-up, and audit replay.

Subject, action, resource, and context schemas for PEP-to-PDP calls — the contracts that make verdict chains replayable.

Golden scenario libraries for PDP/PEP regression, representative, edge, adversarial, and incident replay cases.

Hub for G.A.I.N RAG playbooks: retrieval as a governed action, context packs, and PEP-gated search.

Retrieval as a PEP-gated tool, context pack logging, validation handoff, and PGAR applied to RAG.

Field dictionary for route table rows: route, activation target (which agentic app to start), manifest, policy, retrieval, memory, workflow, and related artifact ids.

Where route contracts live, how to version and promote them, entitlement requirements per row, and rollback in regulated environments.

Golden intent sets, release gates, adversarial coverage, and incident replay for Plane ① routing — aligned with Eval Input plane.

Hub for Policy-Governed Agent Runtime playbooks (foundation, assurance, and boundary). PEP/PDP on the request path.

Instrument applications with latency, traffic, errors, saturation, distributed traces, and SLIs/SLOs. Service-layer observability for unified enterprise stacks.

Two pins, durable run store, and when a checkpointer or Temporal is enough: loop bounds, credential stripping, restart by run_id.

STEP_UP verdict handling, four-eyes approval, re-evaluation with context.approval, and UX ownership in the agentic app.

How to generate synthetic eval cases for edge and adversarial coverage — without polluting golden datasets or optimizing for the generator.

What stays in the agentic app, what the LLM sees, and the PGAR test for credential isolation.

Decide-only router: freeze the route, async-start the app, skip the loop on clarify/abstain. UI and event ingress.