> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Unified Eval Loop > Carry one production failure through monitoring, review, golden coverage, experimentation, deployment, online verification, and signed evidence. # Unified Eval Loop The Unified Eval Loop is the organization-scoped remediation control plane at `/eval-loop`. It operationalizes the improvement flywheel: **mine** production failures, turn them into durable **evals**, **improve** through Playground and Experiments, then **rerun** online verification and seal a signed report. It does not manufacture a success story from aggregate counts or demo data. You start from a real production trace, then advance one persisted workflow only when the referenced platform records prove the next stage. A completed stage is immutable. The workflow version increases by exactly one per accepted transition, and the final state is reached only after a valid signed report records the workflow ID. ## Permissions | Action | Required access | | -------------------------------- | ----------------------------------------------------------------------------------------------- | | View a workflow and its evidence | Organization access and `eval:read` | | Create a workflow | `eval:write` | | Validate and advance a stage | `eval:write` plus permission to create or operate the referenced resource on its native surface | | Delete an untouched workflow | Organization admin and `eval:write` | Identifiers are resolved inside the current organization. A trace, decision, artifact, deployment, or report from another organization is treated as missing; its details are not disclosed. ## The 14-stage evidence chain | Stage | Required canonical evidence | Acceptance rule | | ----------------------- | ---------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Production trace | Trace database ID and immutable trace key | The trace belongs to the organization. This becomes the source identity for every later check. | | Monitor sample | Monitor ID and immutable version, sample ID, monitor-run ID, policy-snapshot hash | The sample and run point to the source trace and the same frozen monitor version. | | Scorer result | Scorer-result ID and version, Model Gateway call ID, cost-record ID, result hash | The successful result is for the source trace. Its Gateway call and authoritative cost record are mutually linked and have complete provenance. | | Alert and failure topic | Alert, topic, topic-version, and membership IDs | One active membership joins the source trace and monitor sample to the exact alert and topic version. | | Review item | Review-item ID | The item is open or assigned and its source or redacted provenance names the workflow alert. | | Candidate case | Candidate ID | The reviewable candidate preserves the original production trace in its source lineage. | | Human approval | Review-item and decision IDs plus rationale | The immutable decision is approved, the review is resolved, and the candidate records the same decision, reviewer, status, and rationale. | | Active golden gate | Evaluation, active test-case, and gate-run IDs plus gate-evidence hash | The approved candidate is promoted to the exact active case and a complete run passes every included case. | | Playground reproduction | Playground, run-snapshot, and winner IDs plus request hash | A completed run uses the golden evaluation and the selected winner names that exact run. | | Experiment winner | Experiment, decision, and winning-variant IDs plus comparison hash | The experiment was created from that Playground, completed, and has an accepted attributable winner decision. | | Published artifact | Artifact and immutable version IDs plus content hash | The selected or live version contains a deployable runtime derived from the winning provider, model, prompt, and parameters. Immutable metadata retains the full winning configuration plus the exact experiment, decision, comparison, and proposal. | | Active deployment | Deployment and artifact-version IDs plus deployment revision | Production serves the exact published version at the recorded revision. | | Online verification | Verification trace and monitor-run IDs, Gateway call, cost record, and policy hash | Fresh traffic passes the original monitor version with no missing evidence and complete Gateway/cost provenance. | | Signed report | Shared-report ID, signing-key version, and report-evidence hash | The non-expired, non-revoked report covers the golden run, names the workflow, and seals the evidence body. | ## Operate the workflow ### 1. Start from a production trace Open `/eval-loop`, choose **Start from a trace**, enter a descriptive workflow name and the numeric trace ID, then select **Create evidence chain**. The source trace is captured as version 1. An API failure leaves the page in an explicit error state; it never substitutes demonstration records. ### 2. Create evidence on its native surface The right-hand panel names the next legal stage, the flywheel phase (Mine / Eval / Improve / Rerun), and a deep link into the workspace that produces that evidence. Use those linked platform workspaces to produce the next record: * Continuous Eval Studio creates and activates the immutable monitor version, samples the trace, runs its checks, and emits the alert. * Scorer Studio executes the exact scorer version through Model Gateway. * Failure Topics records the alert membership; Review Queue records the human decision; candidate promotion creates the active golden case. * Playground and Experiments produce the run snapshot, winner, accepted comparison, and selected release artifact. In Experiments, choose **Publish deployable artifact** (`release_artifact`) so Deployable Assets can approve the exact winning runtime. A release-gate proposal alone cannot advance the Eval Loop `artifact_published` stage. * Deployable Assets approves and deploys the exact version. Continuous Eval then evaluates fresh verification traffic. * Reports signs the final evaluation-run evidence via `POST /api/reports` with `workflowId`. The Eval Loop does not bypass native approval, budget, policy, or evidence requirements. It validates their canonical records and connects them into one versioned chain. ### 3. Review suggested evidence When a workflow has a next legal stage, the panel automatically calls `GET /api/eval-loop/workflows/{workflowId}/suggestions` and pre-fills fields from live organization records linked to the source trace. Use **Refresh suggestions** after you create evidence on a native workspace. Suggestions never invent IDs; they only propose what already exists. Suggestion status: * `ready`: every required field for the next stage was found — review and validate. * `incomplete`: some evidence is still missing. Create it on the linked workspace, then refresh suggestions. Partial fields may still be pre-filled. * `complete`: the workflow has no further stage to suggest. Operators can edit any suggested field. **Validate and advance** always runs full server-side validation against live canonical records. ### 4. Validate the next stage The right-hand panel shows only the next legal stage. Confirm or edit the suggested IDs and hashes, then select **Validate and advance**. The server checks organization ownership, expected workflow version, stage order, and cross-resource lineage before it writes an append-only transition. If validation fails, correct the evidence on its native surface or refresh the workflow. A rejected request does not advance the version or partially persist the stage. ### 5. Verify completion After online verification passes, create a signed report for the golden evaluation and gate run with `POST /api/reports`. Include `workflowId` in that request. The Reports API accepts it only when the workflow belongs to the current organization, is active at `online_verified`, and names the same golden evaluation and gate run. It seals the workflow ID, source trace, version, stage, and evidence hash into the signed body. Reports that are not part of an Eval Loop remain backward compatible and do not require `workflowId`. Use the persisted report ID, signing-key version, and signed-body hash for the `report_signed` transition. When that transition succeeds, the workflow becomes `completed`; neither its source identity nor transition history can be changed. ## API contract | Operation | Purpose | | -------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `POST /api/eval-loop/workflows` | Create version 1 from an organization-owned trace. | | `GET /api/eval-loop/workflows/{workflowId}` | Read the current stage, version, status, and accumulated evidence. | | `GET /api/eval-loop/workflows/{workflowId}/suggestions` | Propose next-stage evidence from live organization records linked to the source trace. | | `POST /api/eval-loop/workflows/{workflowId}/transitions` | Validate the one legal next stage against live canonical records. | | `DELETE /api/eval-loop/workflows/{workflowId}` | Admin-only removal of an untouched active version-1 workflow. | | `POST /api/reports` | Sign the exact online-verified workflow and golden gate when `workflowId` is supplied. | Create requests and transition requests require an idempotency key in the JSON body. Replaying the same key and identical request returns the original result, including under concurrent delivery. Reusing a key for different content fails with `IDEMPOTENCY_CONFLICT`. Every transition includes `expectedVersion`. If another operator advances the workflow first, the stale request fails with `STALE_VERSION`; refresh and inspect the accepted evidence before deciding whether another action is still needed. ## Failure and recovery behavior | Error | Meaning | Recovery | | ---------------------- | ---------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | `NOT_FOUND` | The workflow or referenced record is absent from the current organization. | Confirm the organization and canonical ID. Do not retry with a cross-organization identifier. | | `INVALID_TRANSITION` | The request skipped, repeated, or reordered a stage, or tried to delete a workflow with accepted evidence. | Refresh the workflow and submit only its displayed next stage. | | `STALE_VERSION` | The workflow advanced after the client loaded it. | Refresh, review the newly accepted evidence, and use the new version only if the stage still needs action. | | `EVIDENCE_NOT_FOUND` | One or more required canonical records do not exist in the organization. | Finish the native workflow and copy the persisted identifiers. | | `EVIDENCE_MISMATCH` | The records exist but do not form the required source-trace lineage. | Inspect trace, version, review, artifact, deployment, and report provenance; never replace the evidence with an unrelated successful record. | | `IDEMPOTENCY_CONFLICT` | A key was reused with different content. | Preserve the original request for a retry or generate a new key for a genuinely new action. | An alert or topic is not considered remediated merely because it is labelled, acknowledged, or linked to a proposed fix. Closure requires the approved golden case, exact deployed artifact, healthy online verification, and signed report. ## Troubleshooting checklist 1. Refresh the workflow and confirm its current version and highlighted stage. 2. Open the native resource and verify that its terminal state is persisted, not only visible in a client-side draft. 3. Compare the source trace, monitor/scorer versions, Model Gateway call, cost record, review decision, artifact version, and deployment revision exactly. 4. For online verification, confirm the run used the original monitor version, detected no failure, and lists no missing evidence. 5. For the report, confirm the signing key is current, the report is not expired or revoked, and the report body includes the workflow ID. ## Verification evidence The implementation is exercised by [state-machine tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/unit/eval-loop/production-state-machine.test.ts), [authenticated route tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/api/eval-loop-workflows.routes.test.ts), [real PostgreSQL lifecycle tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/integration/eval-loop/unified-eval-loop.db.test.ts), [DOM tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/dom/pages/eval-loop-page.test.tsx), and the [non-seeded browser journey](https://github.com/evalgate/ai-evaluation-platform/blob/main/e2e/eval-loop-golden-path.spec.ts).