​

Current-behavior manuals

These manuals describe behavior in the 3.7.0 release candidate contract, last verified 2026-07-13. Check Feature status before adopting a surface. Unless a section says otherwise, every resource is organization-scoped and requires authentication; an identifier from another organization returns not found or forbidden rather than exposing the resource.

​

Unified Eval Loop

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Reading requires eval:read; creation and transitions require eval:write; deleting an untouched workflow requires an organization admin.
• UI entry point: /eval-loop starts from a real production trace and displays one organization-scoped, versioned remediation chain without demo fallback.
• API: Four authenticated /api/eval-loop/workflows* operations create, read, advance, and delete an untouched workflow. Native Continuous Eval, Scorer, Failure Topic, adjudication, Playground, Experiment, artifact, deployment, and report operations create the evidence being validated.
• Evidence semantics: Fourteen ordered stages preserve the source trace and exact monitor/scorer versions, Gateway/cost links, alert/topic membership, human decision, active golden case, Playground/Experiment winner, immutable deployable artifact version with the full winning configuration in metadata, deployment revision, online verification, and signed report.
• Failure behavior: Stale, skipped, partial, foreign, mismatched, expired, revoked, or provenance-incomplete evidence does not advance the workflow. Idempotent retries return the original result; a conflicting key fails explicitly.
• Manual: See Unified Eval Loop for the complete stage contract, operator workflow, API, evidence rules, and recovery guidance.

Verification: state machine, authenticated API, PostgreSQL lifecycle, DOM, and non-seeded browser proof.

​

Playground

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Organization members can inspect a Playground. Mutating variants, running cases, accepting proposals, or selecting a winner requires the corresponding evaluation write permission; winner selection is admin-only.
• UI entry point: Open /playgrounds, select an evaluation, then use /evaluations/{id}/playground.
• API: /api/playgrounds lists available workspaces. /api/evaluations/{id}/playground/* manages variants, cases, rows, runs, cancellation, scorers, proposals, the selected winner, and export. The operations are represented in the 3.7.0 release-candidate OpenAPI contract; dedicated generated-client methods are still absent.
• SDK/CLI: No dedicated Playground client. The CLI can produce evaluation inputs and reports, but cannot reproduce the full variant-to-winner workflow.
• Configuration: Create or import an evaluation, add cases, configure provider credentials and an available model, then create at least one prompt variant. A baseline variant is required for meaningful comparison.
• Limitations: Controlled-beta surface. Provider execution depends on Model Gateway health and budget. Generated proposals are drafts until explicitly accepted. Do not treat a UI-only run as a release gate.
• Evidence semantics: A run records its evaluation, variant, case rows, scorer outputs, model-call references, cost state, and terminal status. Cancelled, partial, or provenance-incomplete runs are not verified winners.
• Failure behavior: Validation errors do not start a run. Provider, timeout, or budget failures remain visible per row; cancellation is terminal and must not be reported as success. A duplicate mutation uses its idempotency contract where supported.
• Troubleshooting: Confirm the evaluation has cases and a baseline, the provider is healthy, the requested model is synchronized, and the organization budget allows the call. Inspect failed rows and Model Gateway call detail before retrying.

Verification: route tests, database tests, and UI tests.

​

Prompt Hub

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Read/resolve requires eval:read; authoring and review submission require eval:write; Gateway tests require runs:write; approval, protected publish, and rollback require prompt:publish.
• UI entry point: /prompts opens the organization registry and /prompts/{promptId} opens the complete version workbench.
• Behavior: Stable prompt keys contain immutable content-hashed message versions. Protected environment aliases resolve approved exact versions without draft fallback. Review and alias changes use revisions and append-only history.
• Evidence semantics: Playground, experiment, trace, deployment, evaluation, online-eval, model-call, and Evals-as-Code consumers retain the exact versionId and contentHash; unknown cost remains explicit.
• Manual: See Prompt Hub for complete authoring, schema, tool, Gateway test, review, production, rollback, attribution, API, and troubleshooting behavior.

Verification: API routes, database invariants, workbench DOM, and authenticated golden path.

​

Scorer Studio

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Reads require eval:read; authoring, examples, review submission, and consumer bindings require eval:write; tests and alignment require runs:write; calibration approval, version approval, and environment publishing require scorer:publish.
• UI entry point: /scorers opens the registry and /scorers/{scorerId} opens the evidence-dense Definition, Examples, Test, Human alignment, Calibration, Versions, and Usage workbench.
• Behavior: Deterministic, code, composite, and LLM judge definitions become immutable content-hashed versions. LLM calls use the Model Gateway. Composite components and consumer bindings reference exact version IDs and hashes.
• Execution security: The local TypeScript/Python runner is restricted to trusted non-production development. Production has no local bypass: it requires the configured Docker/Podman OCI boundary, digest-pinned runtimes, the integrity-checked default-deny seccomp profile, network denial, read-only non-root execution, secret isolation, and resource ceilings.
• Limitations: No dedicated SDK or CLI workflow exists yet.
• Evidence semantics: Results retain raw score, withheld-or-mapped normalized score, comparability state, exact scorer version/hash, surface/resource, model-call ID, calibration IDs, logs, cost, latency, and explicit failure state.
• Manual: See Scorer Studio for complete authoring, code limits, examples, alignment/confusion, calibration, approval, exact reuse, API, and troubleshooting behavior.

Verification: unit tests, non-skipped hostile-code OCI proof, API routes, database invariants, workbench DOM, and authenticated golden path.

​

Logs & Trace Explorer

Status: Beta · Owner: Production Intelligence · Last verified: 3.7.1 (2026-07-21)

• Permissions: Query, detail, aggregation, and comparison require traces:read; saved-view, cohort, tag, review, and workflow mutations require their matching trace/evaluation write scopes. Raw retained payloads require organization-admin authority and admin:org.
• UI entry point: /logs opens the virtualized investigation workspace.
• API: /api/logs/* provides canonical queries, stable cursors, per-collection detail continuation, exact bounded aggregation, saved views, cohorts, bulk actions, and audited CSV/JSONL export.
• SDK/CLI: No complete dedicated client exists; use authenticated REST. Safe share URLs preserve filters without transferring raw-data authority.
• Configuration and limits: Queries allow at most 20 structured filters and one primary cursor-safe sort. Pages contain at most 100 traces; detail collections continue independently after 100 records; exports stop at 5,000 rows or 5 MiB.
• Evidence semantics: Rows join authoritative traces, spans, tools, model-call ledger, costs, scores, policy, online alerts, and review records. Unknown cost and incomplete provenance remain explicit; aggregates are exact or fail with QUERY_TOO_COMPLEX.
• Failure behavior and troubleshooting: Invalid fields, sorts, cursors, raw permission, or organization bindings fail closed. Reopen the canonical query, reduce its bounded population, follow collection cursors, and inspect the audit ID for export failures.
• Manual: See Logs & Trace Explorer for the query grammar, retention/redaction contract, keyboard workflow, action receipts, endpoint scopes, and recovery guidance.

​

EvalGate Copilot

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Reading threads/proposals requires eval:read; asking, accepting, and rejecting require organization membership and eval:write. Acceptance rechecks current authority instead of trusting the generation-time snapshot.
• UI entry point: /copilot, with retained context initiated from supported Prompt, Dataset, Scorer, Monitor, Dashboard, Deployment, and Evals-as-Code surfaces.
• API and clients: /api/copilot/* manages threads, retained redacted context, messages, Gateway actions, proposal review, accept/apply-to-draft, and reject. No dedicated SDK/CLI workflow exists.
• Configuration and limits: A thread retains at most 12 resources and bounds each model-context snapshot to 32 KiB. Blocking truncation, redaction, stale hash, expiration, permission loss, or an unaccepted cost bound prevents mutation.
• Evidence semantics: Every proposal preserves redacted before/after state, evidence references, rationale, warnings, expected cost, Model Gateway call ID, authorization snapshots, decision actor/rationale, and created draft/rollback source.
• Failure behavior and troubleshooting: Copilot never publishes or silently changes protected state. STALE_PROPOSAL, budget, permission, partial-application, and validation outcomes remain attributable; regenerate from current context or finish the protected transition in its domain workspace.
• Manual: See EvalGate Copilot for supported resource behavior, prompt-injection controls, proposal fields, and recovery.

​

Remote Runners

Status: Beta · Owner: Evaluation Runtime · Last verified: 3.7.1 (2026-07-21)

• Permissions: Users need runs:read to inspect and runs:write to enqueue/cancel. Registration and runner lifecycle require organization-admin authority; worker routes accept only short-lived runner credentials with matching runner:* scopes.
• UI entry point: /remote-runners manages identities, capacity, drain/revoke, jobs, claims, streams, cancellation, loss, retries, completion, and reconciliation.
• API and clients: /api/remote-runners/* is the signed claim-once protocol. TypeScript and Python SDK workers implement canonical hashing, signature/expiry/binding verification, credential refresh, heartbeat, streaming, cancellation, and completion; no complete operator CLI exists.
• Configuration: Every job binds an immutable release/version/content hash, canonical environment, capability/resource/network policy, and idempotency key. Lease recovery runs through the scheduled job runner or its scoped admin endpoint.
• Evidence semantics: Accepted completion links the exact evaluation run, test results, model calls, costs, trace, stream checksum, claim attempt, duplicate/stale decisions, and one terminal result.
• Failure behavior and troubleshooting: Capability mismatch does not relax policy. Loss expires the lease and creates a new attempt; late, duplicate, malformed, cross-run, and cross-organization completions fail or reconcile explicitly without duplicate results or charges.
• Manual: See Remote Runners for protocol envelopes, SDK examples, isolation policy, scheduled recovery, and incident handling.

​

Deployable Assets

Status: Beta · Owner: Evaluation Runtime · Last verified: 3.7.1 (2026-07-21)

• Permissions: Listing requires runs:read; governed invocation requires runs:write; asset creation, approval, protected rollout, health evidence, pause/resume, promotion, and rollback require organization-admin authority and admin:org.
• UI entry point: /deployable-assets manages approved prompt, scorer, tool, workflow, and agent-entrypoint versions by environment.
• API and clients: /api/deployable-assets exposes list/create, approval, transition, rollout, health, and invocation. Generated contract types are available; dedicated TypeScript, Python, and CLI deployment workflows remain incomplete.
• Configuration: Protected environments resolve only an approved exact version/content hash. Input/output schema, auth, data policy, rate, budget, Gateway execution, deterministic canary weight, non-serving shadow, and health thresholds are evaluated before promotion.
• Evidence semantics: Each invocation records exact asset/deployment/version/environment, route decision, Model Gateway call, cost/policy/trace references, schema result, and rollout health. Unknown cost remains unknown.
• Failure behavior and troubleshooting: Stale/unapproved versions, paused deployments, exhausted rate/budget, policy/schema failure, unhealthy canary, or missing evidence fail closed. Promotion serializes; rollback atomically restores the prior serving version.
• Manual: See Deployable Assets for rollout states, invocation evidence, canary/shadow comparison, promotion, pause/resume, and rollback.

​

Multi-agent Trajectory Analysis

• Purpose: Reconstruct, score, compare, and report hosted or remote multi-agent runs with exact participant, parent, tool, model-call, cost, and environment evidence.
• Primary UI: /trajectory-analysis
• Primary API: /api/trajectory-analysis
• Completion rule: Partial and malformed streams remain explicit and cannot be scored or reported; EvalGate never synthesizes missing evidence.
• Manual: See Multi-agent Trajectory Analysis for the versioned contract, large and late ingestion, evidence integrity, scoring, reports, comparison, retention, and incident handling.

​

Red-Team Workbench

• Capability: Versioned risk packs and reviewed attack cases, immutable target/version/hash campaign snapshots, bounded Model Gateway execution, stable finding dedupe, attributable human transitions, rerun comparison, and a seven-link remediation-to-signed-report chain.
• UI: /red-team
• API: /api/red-team/*
• Permissions: red_team:read, red_team:run, and red_team:approve
• Manual: See Red-Team Workbench for complete risk-pack review, campaign approval, execution/cancellation, finding adjudication, candidate/golden promotion, release-gate linkage, signed evidence, API, and troubleshooting behavior.

Verification: unit contracts, API routes, database invariants, workbench DOM, and authenticated golden path.

​

Dashboards and monitors

Status: Beta · Owner: Evidence & Reporting · Last verified: 3.7.1 (2026-07-21)

• Permissions: Evaluation readers can render published project and organization dashboards. Owners manage their dashboards and shares; metric approval requires an admin. Personal dashboards remain owner-only, and share tokens never bypass organization membership.
• UI entry point: /dashboards opens metric authoring, the versioned dashboard builder, partial evidence-linked rendering, and stateful alert rules. /dashboards/shared/{token} renders the exact immutable version captured by a share link.
• Behavior: Approved, content-hashed metric versions read finite authoritative sources with explicit unknown and calibration semantics. Dashboard/widget changes create immutable versions. Alert transitions retain metric-cache evidence and durable notification/webhook delivery attempts.
• Limitations: Queries fail explicitly above the 10,000-source-row bound. The TypeScript SDK, Python SDK, and CLI do not yet expose the complete workflow.
• Manual: See Dashboards and monitors for supported sources/measures, query completeness, layouts, sharing, alert states, APIs, recovery guidance, and full verification evidence.

Verification: unit, API, database, DOM, and authenticated golden-path suites.

​

Failure Topics

Status: Beta · Owner: Production Intelligence · Last verified: 3.7.1 (2026-07-21)

• Permissions: Trace readers can inspect topics. Discovery, acceptance, human versions, and corrective actions require trace write access; merge and split require an organization admin.
• UI entry point: /insights opens the topic queue and evidence workspace, including impact segments, membership provenance, merge/split controls, human decisions, and downstream coverage links.
• Behavior: Deterministic clustering creates stable failure-mode fingerprints from open, unsuppressed failure reports. Versions, memberships, snapshots, human decisions, idempotent action receipts, and downstream resources are append-only and organization-scoped. Gateway-generated labels stay proposed until a human accepts them.
• Evidence semantics: Deduplicated occurrence counts contribute to recurrence without fabricating membership rows. Cost is known only when priced Model Gateway ledger evidence exists. Merge/split resets trend continuity, and bulk action failures remain itemized as partial.
• Manual: See Failure Topics for discovery thresholds, cohorts, evidence, version controls, AI proposals, merge/split lineage, governed candidate promotion, API operations, and recovery guidance.

Verification: unit, API, database, DOM, and authenticated golden-path suites.

​

Dataset Hub

Status: Beta · Owner: Evaluation Workflows · Last verified: 3.7.1 (2026-07-21)

• Permissions: Members with evaluation read access can list cases. Adding, editing, superseding, deprecating, restoring, archiving, or deleting cases requires the matching write permission.
• UI entry point: Open an evaluation and use its Test Cases, labeled-data, and Synthesize panels. There is no separate global Dataset Hub screen.
• API: /api/evaluations/{id}/test-cases/* manages lifecycle state. /labeled-cases imports and lists reviewed labels. Run-dataset endpoints attach a dataset to a run.
• SDK/CLI: Both SDK workflows use canonical JSONL. CLI label, analyze, cluster, synthesize, and sync commands cover local dataset work; lifecycle administration is not fully client-generated.
• Configuration: Canonical labeled rows need a stable case ID, input, expected/actual values where applicable, label, failure mode, and timestamp. Keep train/development/test partitions separate when calibrating judges or prompts.
• Limitations: Dataset health and coverage signals are advisory. Import does not make a row golden; review and promotion are separate actions. Deletion may be restricted by evidence retention.
• Evidence semantics: Lifecycle transitions retain source and supersession links. Golden evidence must point to the reviewed source, active case version, run, and scorer result. Deprecated or archived cases stay historical but do not silently gate new runs.
• Failure behavior: Invalid rows fail validation. Duplicate identities are deduplicated or rejected; stale updates fail. Partial imports must report accepted and rejected rows rather than claiming complete success.
• Troubleshooting: Validate JSONL, inspect rejected-row reasons, confirm evaluation ownership and case state, and follow supersession links when a historical case no longer appears active.

Verification: golden lifecycle tests and dataset health tests.

​

Experiments

Status: Beta · Owner: Evaluation Runtime · Last verified: 3.7.1 (2026-07-21)

• Permissions: Evaluation readers can inspect runs. Starting, stopping, comparing, or accepting experiment output requires run/auto permissions; privileged acceptance actions may require an admin.
• UI entry point: /evaluations/{id} contains run launch, recent runs, analysis, and Auto experiment panels.
• API: Evaluation run routes start/list runs and expose detail, progress, diff, analysis, clustering, dataset attachment, debug, import, export, and publishing. Auto-session routes plan and execute bounded experiments.
• SDK/CLI: CLI run, auto, diff, gate, and report commands support file/CI experiments. Web Auto orchestration has no complete generated client.
• Configuration: Define cases, scorers, a baseline, provider/model settings, concurrency, budget, and explicit stop conditions. Autonomous modes need an objective and bounded experiment budget.
• Limitations: Auto recommendations are proposals, not releases. Mocked or imported results are identified by source and cannot substitute for provider-backed runtime evidence. Python control-plane parity is incomplete.
• Evidence semantics: Each result is tied to a run, case/version, scorer, baseline or candidate, model-call lineage, and terminal status. Comparisons must disclose missing/partial rows and provenance state.
• Failure behavior: A run can finish failed, cancelled, timed out, budget-blocked, or partial. These states never collapse into passed. Duplicate start requests must use idempotency where the route supports it.
• Troubleshooting: Inspect run progress and per-case errors, confirm provider/cost provenance, compare the same dataset version, and do not publish a run with incomplete evidence.

Verification: experiment query tests, runner tests, and golden flow tests.

​

Continuous Eval

Status: Beta · Owner: Evaluation Runtime · Last verified: 3.7.1 (2026-07-21)

• Permissions: Monitor reads require organization access. Creating, testing, changing, pausing, or resuming a monitor requires an admin and the relevant scopes.
• UI entry point: /online-evals opens Continuous Eval Studio; /online-evals/{monitorId} exposes Overview, Samples, Runs, Alerts, Topics, Candidates, Cost, Configuration, and immutable History for the selected monitor.
• API: /api/online-evals/* covers monitor drafts and versions, historical simulation, explicit activation, pause/resume, bounded backfills with cancel/resume, attributable samples and runs, health/topics/candidates, alert delivery evidence, acknowledgement/resolution, and evidence-gated remediation.
• SDK/CLI: No dedicated TypeScript, Python, or CLI monitor workflow. Use the authenticated REST contract and preserve monitor-version and idempotency identifiers.
• Configuration: Freeze an exact Logs predicate, sampling policy, approved Scorer and Prompt versions, optional published Dataset version, budget and model-call caps, alert delivery channels, and candidate/review behavior. Simulate the immutable draft and accept configured warnings before activation.
• Limitations: Sampling does not guarantee evaluation of every production event. Backfills are bounded to 31 days and 500 candidate traces per request. Provider execution still depends on organization Model Gateway health, and dedicated convenience clients remain incomplete.
• Evidence semantics: Every decision remains attributable to the immutable monitor version and exact trace. Runs link the sampling explanation, configuration hash, scorer/prompt/dataset snapshots, Model Gateway call, authoritative cost and judge records, alerts, channel deliveries, review items, candidates, and remediation attempts. Missing cost or provenance is unknown/incomplete, never zero or verified.
• Failure behavior: Unsupported or stale saved-view predicates fail explicitly. Atomic claim prevents duplicate sample execution; budgets reserve before egress; provider outages, budget blocks, incomplete evidence, failed deliveries, late completion, and retry history remain distinct. A remediation cannot resolve an alert without a successful healthy verification run and matching Experiment, approved Prompt, Model Gateway, cost, and judge evidence.
• Troubleshooting: Inspect version and query hashes, simulation warnings, sampling reason, claim/run state, reservation and authoritative charge totals, provider and provenance health, channel delivery attempts, and remediation evidence. Retry only failed or retryable work with its stable idempotency key; create a new version for changed configuration.

Verification: unit lifecycle tests, authenticated route tests, PostgreSQL lifecycle and remediation tests, concurrent budget tests, Studio DOM tests, and the simulation-to-remediation golden path. The detailed Continuous Eval Studio manual documents the full operator contract.

​

Review Queue

Status: Beta · Owner: Trust & Review · Last verified: 3.7.1 (2026-07-21)

• Permissions: Reviewers see tasks within their organization and assignment. Creating tasks, reassigning, or overriding outcomes requires the corresponding review/admin permission.
• UI entry point: /review; evaluation detail pages expose human-review state.
• API: Evaluation human-review routes list/create tasks and submit reviews. Candidate-specific review endpoints cover synthetic cases.
• SDK/CLI: CLI review supports local labeling/gating workflows. There is no complete generated queue client.
• Configuration: Configure rubric, assignee or reviewer pool, required review count, disagreement handling, and escalation behavior.
• Limitations: Human decisions remain subject to reviewer availability and rubric quality. A single completed label is not consensus unless the configured policy says it is.
• Evidence semantics: Preserve task, item/version, rubric/version, reviewer, decision, reason, timestamps, and supersession. Disagreement and missing-review states must remain visible.
• Failure behavior: Duplicate submissions are idempotent where supported. Stale or already-final tasks reject conflicting updates. Unauthorized reviewers cannot read or submit another organization’s task.
• Troubleshooting: Confirm assignment and role, refresh task state, inspect required-review count and disagreement status, then escalate instead of overwriting a completed review.

Verification: route tests and integrity tests.

​

Synthetic and golden lifecycle

Status: Beta · Owner: Trust & Review · Last verified: 3.7.1 (2026-07-21)

• Permissions: Members can inspect candidates. Review, rejection, replay, and promotion require synthesis/evaluation write permissions; policy may require explicit human review before promotion.
• UI entry point: /candidates and an evaluation’s Synthesize and Test Cases panels.
• API: Candidate list/detail/update, replay, and promote routes; evaluation-scoped synthetic candidate list, review, and promotion routes; test-case lifecycle routes retain the promoted result.
• SDK/CLI: CLI synthesize, replay, and promote; canonical golden-case schemas are available in TypeScript and Python.
• Configuration: Set generation dimensions, deduplication thresholds, PII policy, required provenance, review gate, and destination evaluation/golden suite.
• Limitations: Synthetic cases are quarantined drafts. Diversity scores are heuristics. No candidate becomes trusted evidence merely because a model generated it or a heuristic score is high.
• Evidence semantics: Candidate provenance includes source failures/cases, prompt/model/version, generator configuration, detectors, reviewer decision, and promotion link. Promotion creates or links a lifecycle record without erasing quarantine history.
• Failure behavior: PII or missing provenance blocks promotion. Duplicate candidates are rejected/deduplicated. Replay failure remains a failed replay and cannot auto-promote.
• Troubleshooting: Inspect provenance and detector results, resolve review requirements, compare against active golden cases for duplicates, and replay against the current model before promotion.

Verification: lifecycle DB tests, review-gate tests, and promotion tests.

​

Model Gateway

Status: Beta · Owner: Gateway & Providers · Last verified: 3.7.1 (2026-07-21)

• Permissions: Organization members with read scopes can inspect allowed records. Config, routing, model sync, health checks, and test calls require an admin. Secrets are never returned after write.
• UI entry point: /settings → Model Gateway.
• API: Config CRUD, key rotation, health checks, model sync/list, routing-profile CRUD/test/explain, test call, call ledger/detail, and provenance health under /api/model-gateway/*.
• SDK/CLI: No dedicated generated Model Gateway client or CLI in this release.
• Configuration: Model-backed workflows are BYOK and do not include bundled inference credits. Register an allowed direct provider or gateway URL, store the organization-owned credential, synchronize models, select a default config/routing profile, then pass health and test-call checks. The connected provider bills usage directly. Production URLs and ports are constrained by egress policy. See Model providers and BYOK.
• Limitations: Provider/model availability varies. Unknown pricing stays unknown. Registered custom gateways are revalidated before egress; redirects and private/link-local destinations are blocked.
• Evidence semantics: Every allowed or blocked call has an organization-scoped ledger row and policy decision. Provider attempts, retries/fallback, cost state, evidence, and terminal status are linked. A pre-egress block has no cost row.
• Failure behavior: Policy/budget blocks produce no provider egress. Secret-bearing provider errors are redacted. Timeouts, retries, fallback attempts, duplicate submissions, and incomplete provenance remain distinguishable.
• Troubleshooting: Run health check and model sync, verify the configured model/provider, inspect route explanation and call detail, confirm budget and policy decision, then check provenance health before retrying.

Verification: config DB tests, ledger tests, and registered-egress tests.

​

Evals-as-Code

Status: Beta · Owner: Developer Experience · Last verified: 3.7.1 (2026-07-21)

• Permissions: Manifest reads require organization access. Validation/planning can be non-mutating; apply, rollback, retry, drift acceptance, and remediation require appropriate write/admin scopes.
• UI entry point: CLI and CI are primary. Evaluation pages display resulting evaluations/runs; there is no complete manifest-management dashboard.
• API: Validate, plan, diff, gate, apply, apply status/retry/rollback, manifest/version reads, and drift-event list/accept/remediate under /api/evals-as-code/*.
• SDK/CLI: TypeScript CLI is the primary supported workflow. Python supports core evaluation/runtime pieces but does not yet match every control-plane operation.
• Configuration: Commit the manifest and referenced prompts/datasets, pin the contract version, use a stable idempotency key for applies, and separate plan from mutation in CI.
• Limitations: Applying a manifest changes organization state. Generated plans must be reviewed. Cross-SDK and public-contract parity are still Beta gates.
• Evidence semantics: Canonical manifest hash, plan/diff, actor, apply attempt, resulting resource versions, gate output, drift decision, and rollback link form the audit trail.
• Failure behavior: Validation and stale-version failures do not mutate. Apply is idempotent; partial failures remain visible and can be retried/rolled back according to recorded state. Drift is not silently accepted.
• Troubleshooting: Validate first, compare canonical hashes, inspect apply status and failed operation, reuse the original idempotency key for retry, and roll back only from a recorded reversible apply.

Verification: route tests, idempotency DB tests, and rollback/drift tests.

​

Provider onboarding

Status: Beta · Owner: Gateway & Providers · Last verified: 3.7.1 (2026-07-21)

• Permissions: Provider configuration, credential write, model sync, and test calls require an organization admin. Members can use only providers/models allowed by organization policy.
• UI entry point: /settings → Provider Keys and Model Gateway.
• API: Provider-key administration plus Model Gateway config creation, health check, model sync, and test call are included in the verified public OpenAPI contract and generated TypeScript contract types.
• SDK/CLI: No dedicated onboarding client or CLI. SDK runtime provider configuration is separate from storing organization credentials in the platform.
• Configuration: Bring your own provider key (BYOK): choose a provider type, provide the organization-owned endpoint/credential fields, save, run health check, synchronize models, select a default/routing policy, and complete a real test call. EvalGate does not bundle model inference credits, and a blank organization has no implicit usable provider. The EvalGate API key authenticates platform requests; the provider credential authorizes separately billed model calls.
• Limitations: Supported fields and health semantics vary by provider. Local/private URLs are not production defaults; custom gateway egress remains allowlisted and revalidated.
• Evidence semantics: Setup status distinguishes saved, healthy, models synchronized, default selected, and test call proven. Each failed step retains a safe reason without exposing the credential.
• Failure behavior: Missing or invalid credentials fail closed. Provider outages and malformed responses are classified separately from policy/egress blocks. Secrets are redacted from API, UI, logs, and test output.
• Troubleshooting: Follow the ordered checklist, correct the first failed step, rotate a suspect credential, re-run health/model sync, and inspect the redacted test-call result and gateway ledger.

Verification: provider resolution tests, onboarding route tests, and wizard UI tests.

​

Calibration control plane

Status: Beta · Owner: Trust & Review · Last verified: 3.7.1 (2026-07-21)

• Permissions: eval:read inspects the workspace, diffs, drift, and gate history; eval:write creates anchor versions; runs:write fits mappings, records drift, and evaluates gates. Reviewer approval requires scorer:publish plus organization-admin authority. Overrides require the same human authority and an explicit rationale.
• UI entry point: /calibration opens the anchor registry, mapping registry, version diff, drift and trend-break workspace, and release-gate history. Legacy evaluation-detail and /llm-judge surfaces remain available for judge configuration and proposal workflows.
• Behavior: Human anchor contents and provenance become immutable content-hashed versions. Reviewer-approved isotonic, percentile, or z-score mappings bind one exact anchor version and judge configuration. New runtime normalization resolves only the canonical approved mappingVersionId; recalibration creates a new version instead of rewriting history.
• Comparability: Results are exact, calibrated, or not_comparable. Mapping diffs expose method, scale, function, anchor, and statistical changes. Blocking drift or incompatible anchor/scale contracts creates an explicit trend discontinuity and a durable blocked release-gate explanation.
• Human control: Reviews, drift events, gate decisions, and admin overrides are append-only. An override authorizes one subject while preserving the underlying not_comparable state; it never relabels incompatible evidence as calibrated.
• Scorer integration: Scorer Studio binds the canonical mapping version and derives legacy calibration identifiers from it. Comparable results persist the exact scorer version/hash and mapping version; missing approval or mismatched lineage withholds the normalized score.
• API: /api/calibration/sets, set versions and reviews, mappings and reviews, mapping diffs, drift events, workspace history, and release gates form the canonical control-plane API. Existing evaluation calibration proposals remain supported.
• SDK/CLI and limitations: Judge execution exists in both SDKs, but control-plane administration has no complete SDK or CLI workflow. Confidence depends on human-label quality and sample size; missing evidence remains explicit rather than inferred.
• Manual: See Calibration control plane for anchor fields, mapping math and boundaries, lifecycle transitions, exact evidence, failure contracts, endpoints, troubleshooting, and verification.

Verification: mapping and comparability unit tests, authenticated API tests, database lifecycle and retention tests, accessible workspace DOM tests, and the authenticated release-gate path.

​

Reports

Status: Beta · Owner: Evidence & Reporting · Last verified: 3.7.1 (2026-07-21)

• Permissions: Organization members with report/run read scopes can create or list reports. Key rotation and revocation require privileged administration. A public viewer needs the share token but no account.
• UI entry point: Evaluation Export/Report actions, /reports, and a shared /report/{token} or /r/{token} link.
• API: GET/POST /api/reports, evaluation report/export operations, public verification, and share-token revocation are represented in the verified OpenAPI contract and generated TypeScript contract types. Eval Loop completion supplies workflowId to POST /api/reports; the server accepts only an organization-owned, active online_verified workflow whose golden evaluation and gate run match the report request.
• SDK/CLI: CLI gates emit machine-readable regression reports. There is no complete client for signed report creation, verification, key rotation, or revocation.
• Configuration: Select a completed run/evaluation snapshot, audience and scope, expiry/replay policy, and redaction policy. Signing uses the organization’s active key; historical verification keys remain verify-only until revoked.
• Limitations: A report is a snapshot, not a live dashboard. Public links may be one-time or expire. Unknown, revoked, incomplete-provenance, or tampered evidence is not verified.
• Evidence semantics: Signed claims bind organization, audience, scope, version, replay constraints, snapshot digest, and key ID. An Eval Loop report additionally seals the workflow ID, source trace, version, stage, and evidence hash. Report rows expose provenance completeness and unknown cost instead of substituting zero.
• Failure behavior: Signature, audience, organization, expiry, replay, revocation, or digest failure denies verification. Concurrent one-time consumption permits one success. Export aborts if the audit chain is invalid.
• Troubleshooting: Check token expiry/use/revocation, expected audience and organization, signing key state, snapshot digest, and provenance completeness. Create a new link instead of weakening verification.

Verification: signing keyring tests, public verification tests, and export provenance tests.

​

Deployment

Status: Beta · Owner: Developer Experience · Last verified: 3.7.1 (2026-07-21)

• Permissions: CI uses an organization API key with the least scopes needed to ingest runs, execute/read the gate, and create release evidence. Creating or changing a release requires release/evaluation write access.
• UI entry point: Deployment is CI/release-workflow first; there is no single deployment dashboard. Evaluation and report pages show the evidence produced by a deployment gate.
• API: GET/POST /api/releases plus Evals-as-Code validation, plan, gate, apply, and report operations.
• SDK/CLI: TypeScript CLI provides the primary CI gate, JSON output, baseline handling, and report artifacts. Python parity for the entire release control plane is incomplete.
• Configuration: Pin SDK and manifest versions, store API keys in CI secrets, set baseline and thresholds, use machine-readable output, upload evidence artifacts, and make the gate a required check.
• Limitations: EvalGate records/gates a release workflow; it does not deploy application infrastructure. A passing local mock or skipped CI lane is not production evidence.
• Evidence semantics: Preserve commit SHA, environment, manifest/dataset/prompt hashes, baseline and candidate runs, gate result, actor, timestamps, costs, provenance health, and signed report references.
• Failure behavior: Validation, missing evidence, regression, provider failure, timeout, budget block, or incomplete provenance fails or marks the gate inconclusive according to policy. Zero discovered tests is never a pass.
• Troubleshooting: Run preflight locally, verify exact commit and environment, inspect machine-readable gate output, confirm required CI discovery counts, and follow evidence IDs into the run, model-call, cost, and report records.

Verification: CI guide, golden path tests, and regression gate contract.