> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Continuous Eval Studio

> Build, simulate, activate, backfill, operate, and remediate immutable production evaluation monitors with exact trace and model-call evidence.

Continuous Eval Studio turns production traces into a governed evaluation loop. A monitor starts as a draft, freezes its traffic predicate and evaluation assets into an immutable version, proves expected volume and cost on historical traffic, and becomes eligible for explicit activation. Every sampled trace, model call, judge result, cost record, alert delivery, review item, candidate, and remediation remains attributable to the exact monitor version that produced it.

<Info>
  Continuous Eval uses the same authoritative traces, Logs Explorer predicates, Model Gateway calls, cost records, Scorer Studio versions, Prompt Hub versions, Dataset Hub versions, Review Queue items, notifications, and webhooks as the rest of the platform. It does not maintain a second semantic copy of production evidence.
</Info>

## Operator workflow

<Steps>
  <Step title="Create a draft monitor">
    Open **Continuous Eval** in the platform navigation and choose **New monitor**. Configure traffic, sampling, checks, budget, alerting, and learning. Saving creates a draft and an immutable version; it does not begin sampling production traffic.
  </Step>

  <Step title="Bind exact source and evaluation assets">
    Select an authorized Logs saved view when the monitor should follow a reusable production predicate. Optionally bind a deployment, an approved Prompt Hub version, a published Dataset Hub version, and approved Scorer Studio versions. These values and their content hashes are copied into the monitor version.
  </Step>

  <Step title="Simulate historical traffic">
    Open the draft and choose **Simulate**. Select a bounded window of at most 31 days. Simulation performs no provider calls and sends no alerts. It records matched and sampled trace IDs, exclusions, estimated model calls, cost, alert volume, false-positive review examples, budget preflight, and missing provenance.
  </Step>

  <Step title="Review warnings and activate">
    Inspect the simulation result and frozen configuration. Activation requires the exact version ID and configuration hash, a completed simulation for that version, and explicit acceptance of any configured cost or alert-volume warnings. A stale browser cannot activate a changed draft.
  </Step>

  <Step title="Operate from evidence">
    Use Overview, Samples, Runs, Alerts, Topics, Candidates, Cost, Configuration, and History to understand health and act on failures. Links open the exact trace in Logs Explorer and route attributable work to Review Queue, Candidates, Scorer Studio, and Experiments.
  </Step>

  <Step title="Verify remediation">
    A production alert remains open until a verification trace has a successful monitor run with exact model-call, cost, and judge evidence and the operator links the corrective Experiment and Prompt version. Failed or incomplete verification is retained as evidence and does not resolve the alert.
  </Step>
</Steps>

## Builder reference

### Traffic

Traffic configuration determines the population before sampling.

| Field               | Semantics                                                                 |
| ------------------- | ------------------------------------------------------------------------- |
| Environment         | Exact trace environment match, such as `production`                       |
| Trace source        | Exact source match, such as `sdk`                                         |
| Trace name contains | Case-insensitive name predicate                                           |
| Logs saved view ID  | Freezes the complete authorized canonical Logs query and its SHA-256 hash |
| Deployment ID       | Records the deployment boundary in the frozen source snapshot             |
| Prompt version ID   | Must be an approved Prompt Hub version in the same organization           |
| Dataset version ID  | Must be a published Dataset Hub version in the same organization          |

When a saved view is selected, the monitor freezes both its identifier and canonical predicate. Simulation, live sampling, and backfill execute that frozen predicate through the Logs query planner. Later edits to the saved view do not rewrite an active monitor version. Unsupported query versions, invalid fields or operators, missing predicate evidence, and query-hash mismatches are rejected instead of being narrowed to legacy fields.

Personal views are available only to their owner. Project and organization views remain organization-scoped. A view from another organization is reported as unavailable without disclosing its existence.

### Sampling

| Mode                 | Behavior                                                       |
| -------------------- | -------------------------------------------------------------- |
| Random               | Applies the configured base rate                               |
| Failure only         | Excludes traces without a failure status before rate sampling  |
| Severity weighted    | Raises the effective rate for higher-severity failures         |
| Uncertainty weighted | Raises the effective rate when available evidence is uncertain |
| Adaptive             | Applies the frozen adaptive target and policy snapshot         |

Sampling is deterministic for the tuple of monitor ID, immutable version number, and trace key. A sample records its effective rate, deterministic bucket, trace status, severity hint, uncertainty score, novelty state, daily sample count, inclusion reason, and exact policy version. Replaying the same version and trace produces the same rate decision.

The daily sample cap is evaluated before inclusion. A duplicate live event cannot create a second sample for the same monitor version and trace.

### Checks and calibration

Deterministic checks inspect persisted trace status, output signals, latency, and other available evidence. Model-judged checks use the organization Model Gateway; provider calls never bypass the gateway.

Scorer Studio bindings must reference approved immutable versions in the same organization. The monitor freezes each scorer's version ID, content hash, definition, calibration binding, calibration set and run, score mapping, and comparability state. A model judge without an approved comparable calibration remains visibly `not_comparable`; it is not silently treated as a stable trend metric.

If a judge determines that a detector signal is a false positive, a successful trace may be recorded as healthy while retaining the detector and judge evidence. A hard failed trace status remains a failure.

### Budget and provider behavior

| Limit                | Enforcement                                                                    |
| -------------------- | ------------------------------------------------------------------------------ |
| Daily cost cap       | Serializes monitor reservations against committed plus reserved spend          |
| Per-call cost cap    | Blocks preflight when estimated cost per model call exceeds the limit          |
| Daily model-call cap | Blocks projected call volume above the limit                                   |
| Backfill cost cap    | Uses an execution-scoped reservation ledger, enforced under concurrent workers |

Reservations are created before provider egress and finalized from authoritative cost records. Cancellation and expiry release reserved spend. If a provider completion arrives after local cancellation, its actual cost is still charged and marked as a late completion. Unknown cost remains **Unknown**; it is never displayed as zero. A successful call with unavailable pricing still retains a cost-evidence ID marked `unknown_model`, but that evidence anchor does not contribute a monetary amount to committed spend.

Provider failure behavior can continue deterministic checks, skip the provider check honestly, or fail closed according to the frozen policy. Operational health distinguishes `provider_outage`, `budget_blocked`, `incomplete_evidence`, and `degraded` from `healthy`.

### Alerting and learning

Configure a failure-rate threshold, minimum severity, dedupe window, and one or more channels:

* in-app notification;
* subscribed organization webhook;
* attributable Review Queue item.

Delivery attempts have a stable idempotency key per alert and channel. Successful delivery is not repeated. Failed attempts increment durable attempt count and retry history; a missing webhook subscription or review item is shown as a failed channel rather than a successful alert.

Confirmed failures can create deduplicated candidate evaluation cases. Candidates keep the source trace, failure span, generator version, rationale, review linkage, and monitor evidence. They remain quarantined until the configured human-review and promotion policy is satisfied.

## Simulation and activation

Simulation is a persisted execution with `runType: simulation`. It sorts historical traces deterministically, executes the exact frozen source predicate, applies the same sampling rules used by live traffic, and records:

* matched and sampled trace database IDs;
* exclusion counts by reason;
* estimated model calls and cost;
* expected alerts and historical failures;
* a bounded false-positive review set;
* budget preflight reasons and remaining limits;
* prompt, dataset, scorer, policy, and configuration hashes;
* missing evidence and warnings.

Simulation does not call a provider, deliver alerts, create reviews, or generate candidates. Activation fails if there is no completed simulation for the version, the supplied configuration hash is stale, or a required warning has not been accepted.

Activating a new version retires the previous version for new traffic. Existing samples, runs, alerts, deliveries, and remediations continue to reference the version that produced them. Pausing stops new live sampling without deleting evidence. Resume is available only to a paused monitor that already has an active version; a never-activated draft cannot bypass simulation by using resume.

## Run evidence and failure states

Every claimed sample creates a durable run before evaluation starts. Only one worker can claim a pending or retryable sample. A completed sample replay returns the persisted result instead of invoking the provider again.

A run stores:

* monitor version ID, number, configuration hash, and policy snapshot;
* sampled trace ID and key;
* exact scorer, prompt, and dataset snapshots;
* Model Gateway call ID;
* judge-result and cost-record IDs;
* reservation and actual cost;
* detector and judge decision summary;
* alert ID, missing evidence, attempt number, and retry history;
* start and completion timestamps.

| State                 | Meaning                                                  | Recovery                                                                  |
| --------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------- |
| `success`             | Checks completed; inspect `failureDetected` and evidence | Triage an alert or use as verification evidence                           |
| `provider_outage`     | Provider call failed or timed out                        | Inspect provider health, then retry the failed sample                     |
| `budget_blocked`      | A reservation would exceed the applicable cap            | Raise the approved cap or wait for the next budget window                 |
| `incomplete_evidence` | Run completed but required provenance is absent          | Bind or restore the missing exact asset/evidence and create a new version |
| `failed`              | Local execution failed before a terminal result          | Inspect retry history and retry                                           |

Missing model call, cost, judge, prompt, dataset, scorer, or terminal result is listed explicitly. The UI never converts missing evidence into a successful verification claim.

## Backfills

Backfills use an explicit immutable monitor version and a window of at most 31 days and 500 candidate traces per request. Simulation and preflight occur before jobs are enqueued. A backfill has its own execution, progress counters, sampled trace IDs, cost estimate, actual cost, retry history, and cancellation state.

Alert delivery and candidate generation are disabled by default for backfills. Enable them explicitly when historical results should enter production triage. This prevents an exploratory backfill from flooding live alert channels.

Cancellation marks pending samples as cancelled and releases active reservations. In-flight provider completions remain chargeable. Completed evidence is retained. Cancelled, failed, or partially completed backfills can be resumed with a new idempotency key; the resumed execution does not rewrite the original history.

## Alert triage and remediation

An alert records exact monitor, version, execution, sample, run, trace, model call, severity, category, dedupe key, delivery state, review item, candidate, and remediation state. Topic summaries are derived from persisted alert evidence and show count, open count, severity, and alert IDs.

To resolve through remediation, provide:

1. a verification trace ID evaluated by the same monitor;
2. the Experiment ID that proves the corrective comparison;
3. the approved Prompt Hub version ID being promoted;
4. a rationale and idempotency key.

The verification run must be successful, healthy, and contain model-call, cost-record, and judge-result IDs. A judge-confirmed detector false positive can satisfy this condition; a hard failed trace cannot. Cross-organization or nonexistent references are rejected before a remediation record is created. Incomplete verification evidence creates a failed remediation record and leaves the production alert open.

## Permissions

| Operation                                                                      | Minimum access                                                                |
| ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| List monitors, versions, samples, runs, alerts, health, topics, and candidates | `eval:read`                                                                   |
| Simulate, backfill, cancel/resume backfill, and verify remediation             | admin role and `eval:write`                                                   |
| Create, edit, activate, pause, or resume a monitor                             | admin role; activation and monitor lifecycle use `admin:org` where applicable |

All services derive organization and actor from authenticated context. Body or query parameters cannot switch organization scope. Exact Prompt, Dataset, Scorer, Logs view, Experiment, trace, and alert references are rechecked against that organization.

## API reference

| Method         | Endpoint                                    | Purpose                                                                 |
| -------------- | ------------------------------------------- | ----------------------------------------------------------------------- |
| `GET`, `POST`  | `/api/online-evals/monitors`                | List monitors or save a draft                                           |
| `GET`, `PATCH` | `/api/online-evals/monitors/:id`            | Read a monitor or create a new draft version from configuration changes |
| `GET`          | `/api/online-evals/monitors/:id/versions`   | List immutable versions                                                 |
| `POST`         | `/api/online-evals/monitors/:id/simulate`   | Persist a historical no-egress simulation                               |
| `POST`         | `/api/online-evals/monitors/:id/activate`   | Activate an exact simulated version                                     |
| `POST`         | `/api/online-evals/monitors/:id/pause`      | Pause new live sampling                                                 |
| `POST`         | `/api/online-evals/monitors/:id/resume`     | Resume a previously active paused monitor                               |
| `GET`, `POST`  | `/api/online-evals/monitors/:id/backfills`  | List or create bounded backfills                                        |
| `POST`         | `/api/online-evals/backfills/:id/cancel`    | Cancel pending backfill work                                            |
| `POST`         | `/api/online-evals/backfills/:id/resume`    | Resume a terminal retryable backfill                                    |
| `GET`          | `/api/online-evals/monitors/:id/health`     | Read provider, evidence, lifecycle, and budget health                   |
| `GET`          | `/api/online-evals/monitors/:id/topics`     | List evidence-derived alert topics                                      |
| `GET`          | `/api/online-evals/monitors/:id/candidates` | List attributable generated candidates                                  |
| `GET`          | `/api/online-evals/alerts/:id/deliveries`   | Inspect channel attempts and retry history                              |
| `POST`         | `/api/online-evals/alerts/:id/remediation`  | Verify corrective evidence and conditionally resolve                    |

### Draft example

```json theme={null}
{
  "name": "Production checkout quality",
  "matchRules": { "environment": "production", "source": "sdk" },
  "sampling": {
    "mode": "severity_weighted",
    "sampleRate": 0.1,
    "maxDailySamples": 500
  },
  "configuration": {
    "savedViewId": "3c69e167-8ad4-4f68-8bc9-34ea8c0bde21",
    "promptVersionId": 42,
    "datasetVersionId": "166bc5b7-fad1-46d6-b40b-c9e40db9705a",
    "checks": { "deterministic": true, "llmJudge": true, "scorerVersionIds": [18] },
    "budget": {
      "maxDailyCostUsd": 25,
      "maxPerCallCostUsd": 0.1,
      "maxDailyModelCalls": 500,
      "backfillMaxCostUsd": 10,
      "fallbackBehavior": "deterministic_only"
    },
    "alerting": {
      "failureRateThreshold": 0.1,
      "dedupeWindowMinutes": 60,
      "channels": ["in_app", "review_queue"]
    },
    "learning": { "generateCandidates": true, "autoCreateReview": true }
  },
  "activateImmediately": false
}
```

### Simulation and activation examples

```json theme={null}
{
  "versionId": "ddfb2b25-7286-4a70-afc0-a8624c514a62",
  "windowStart": "2026-07-06T00:00:00.000Z",
  "windowEnd": "2026-07-13T00:00:00.000Z",
  "limit": 200,
  "idempotencyKey": "checkout-simulation-2026-07-13"
}
```

```json theme={null}
{
  "versionId": "ddfb2b25-7286-4a70-afc0-a8624c514a62",
  "expectedConfigHash": "a91c52ed8b7e46a4c6bdf302d868d836f7d0ce77c232c1bd0f9f165b03dbf9eb",
  "acceptCostWarning": true,
  "acceptAlertVolumeWarning": true
}
```

<Warning>
  Do not automate activation by setting `activateImmediately`. Production activation is intentionally gated by a persisted simulation and explicit warning acknowledgement.
</Warning>