Unified Eval Loop
The Unified Eval Loop is the organization-scoped remediation control plane at/eval-loop. It operationalizes the improvement flywheel: mine production
failures, turn them into durable evals, improve through Playground and
Experiments, then rerun online verification and seal a signed report. It does
not manufacture a success story from aggregate counts or demo data. You start
from a real production trace, then advance one persisted workflow only when the
referenced platform records prove the next stage.
A completed stage is immutable. The workflow version increases by exactly one
per accepted transition, and the final state is reached only after a valid
signed report records the workflow ID.
Permissions
Identifiers are resolved inside the current organization. A trace, decision,
artifact, deployment, or report from another organization is treated as missing;
its details are not disclosed.
The 14-stage evidence chain
Operate the workflow
1. Start from a production trace
Open/eval-loop, choose Start from a trace, enter a descriptive workflow
name and the numeric trace ID, then select Create evidence chain. The source
trace is captured as version 1. An API failure leaves the page in an explicit
error state; it never substitutes demonstration records.
2. Create evidence on its native surface
The right-hand panel names the next legal stage, the flywheel phase (Mine / Eval / Improve / Rerun), and a deep link into the workspace that produces that evidence. Use those linked platform workspaces to produce the next record:- Continuous Eval Studio creates and activates the immutable monitor version, samples the trace, runs its checks, and emits the alert.
- Scorer Studio executes the exact scorer version through Model Gateway.
- Failure Topics records the alert membership; Review Queue records the human decision; candidate promotion creates the active golden case.
- Playground and Experiments produce the run snapshot, winner, accepted
comparison, and selected release artifact. In Experiments, choose
Publish deployable artifact (
release_artifact) so Deployable Assets can approve the exact winning runtime. A release-gate proposal alone cannot advance the Eval Loopartifact_publishedstage. - Deployable Assets approves and deploys the exact version. Continuous Eval then evaluates fresh verification traffic.
- Reports signs the final evaluation-run evidence via
POST /api/reportswithworkflowId.
3. Review suggested evidence
When a workflow has a next legal stage, the panel automatically callsGET /api/eval-loop/workflows/{workflowId}/suggestions and pre-fills fields from
live organization records linked to the source trace. Use Refresh suggestions
after you create evidence on a native workspace. Suggestions never invent IDs;
they only propose what already exists.
Suggestion status:
ready: every required field for the next stage was found — review and validate.incomplete: some evidence is still missing. Create it on the linked workspace, then refresh suggestions. Partial fields may still be pre-filled.complete: the workflow has no further stage to suggest.
4. Validate the next stage
The right-hand panel shows only the next legal stage. Confirm or edit the suggested IDs and hashes, then select Validate and advance. The server checks organization ownership, expected workflow version, stage order, and cross-resource lineage before it writes an append-only transition. If validation fails, correct the evidence on its native surface or refresh the workflow. A rejected request does not advance the version or partially persist the stage.5. Verify completion
After online verification passes, create a signed report for the golden evaluation and gate run withPOST /api/reports. Include workflowId in that
request. The Reports API accepts it only when the workflow belongs to the current
organization, is active at online_verified, and names the same golden
evaluation and gate run. It seals the workflow ID, source trace, version, stage,
and evidence hash into the signed body. Reports that are not part of an Eval Loop
remain backward compatible and do not require workflowId.
Use the persisted report ID, signing-key version, and signed-body hash for the
report_signed transition. When that transition succeeds, the workflow becomes
completed; neither its source identity nor transition history can be changed.
API contract
Create requests and transition requests require an idempotency key in the JSON
body. Replaying the same key and identical request returns the original result,
including under concurrent delivery. Reusing a key for different content fails
with
IDEMPOTENCY_CONFLICT.
Every transition includes expectedVersion. If another operator advances the
workflow first, the stale request fails with STALE_VERSION; refresh and inspect
the accepted evidence before deciding whether another action is still needed.
Failure and recovery behavior
An alert or topic is not considered remediated merely because it is labelled,
acknowledged, or linked to a proposed fix. Closure requires the approved golden
case, exact deployed artifact, healthy online verification, and signed report.
Troubleshooting checklist
- Refresh the workflow and confirm its current version and highlighted stage.
- Open the native resource and verify that its terminal state is persisted, not only visible in a client-side draft.
- Compare the source trace, monitor/scorer versions, Model Gateway call, cost record, review decision, artifact version, and deployment revision exactly.
- For online verification, confirm the run used the original monitor version, detected no failure, and lists no missing evidence.
- For the report, confirm the signing key is current, the report is not expired or revoked, and the report body includes the workflow ID.