Continuous Eval uses the same authoritative traces, Logs Explorer predicates, Model Gateway calls, cost records, Scorer Studio versions, Prompt Hub versions, Dataset Hub versions, Review Queue items, notifications, and webhooks as the rest of the platform. It does not maintain a second semantic copy of production evidence.
Operator workflow
1
Create a draft monitor
Open Continuous Eval in the platform navigation and choose New monitor. Configure traffic, sampling, checks, budget, alerting, and learning. Saving creates a draft and an immutable version; it does not begin sampling production traffic.
2
Bind exact source and evaluation assets
Select an authorized Logs saved view when the monitor should follow a reusable production predicate. Optionally bind a deployment, an approved Prompt Hub version, a published Dataset Hub version, and approved Scorer Studio versions. These values and their content hashes are copied into the monitor version.
3
Simulate historical traffic
Open the draft and choose Simulate. Select a bounded window of at most 31 days. Simulation performs no provider calls and sends no alerts. It records matched and sampled trace IDs, exclusions, estimated model calls, cost, alert volume, false-positive review examples, budget preflight, and missing provenance.
4
Review warnings and activate
Inspect the simulation result and frozen configuration. Activation requires the exact version ID and configuration hash, a completed simulation for that version, and explicit acceptance of any configured cost or alert-volume warnings. A stale browser cannot activate a changed draft.
5
Operate from evidence
Use Overview, Samples, Runs, Alerts, Topics, Candidates, Cost, Configuration, and History to understand health and act on failures. Links open the exact trace in Logs Explorer and route attributable work to Review Queue, Candidates, Scorer Studio, and Experiments.
6
Verify remediation
A production alert remains open until a verification trace has a successful monitor run with exact model-call, cost, and judge evidence and the operator links the corrective Experiment and Prompt version. Failed or incomplete verification is retained as evidence and does not resolve the alert.
Builder reference
Traffic
Traffic configuration determines the population before sampling.
When a saved view is selected, the monitor freezes both its identifier and canonical predicate. Simulation, live sampling, and backfill execute that frozen predicate through the Logs query planner. Later edits to the saved view do not rewrite an active monitor version. Unsupported query versions, invalid fields or operators, missing predicate evidence, and query-hash mismatches are rejected instead of being narrowed to legacy fields.
Personal views are available only to their owner. Project and organization views remain organization-scoped. A view from another organization is reported as unavailable without disclosing its existence.
Sampling
Sampling is deterministic for the tuple of monitor ID, immutable version number, and trace key. A sample records its effective rate, deterministic bucket, trace status, severity hint, uncertainty score, novelty state, daily sample count, inclusion reason, and exact policy version. Replaying the same version and trace produces the same rate decision.
The daily sample cap is evaluated before inclusion. A duplicate live event cannot create a second sample for the same monitor version and trace.
Checks and calibration
Deterministic checks inspect persisted trace status, output signals, latency, and other available evidence. Model-judged checks use the organization Model Gateway; provider calls never bypass the gateway. Scorer Studio bindings must reference approved immutable versions in the same organization. The monitor freezes each scorer’s version ID, content hash, definition, calibration binding, calibration set and run, score mapping, and comparability state. A model judge without an approved comparable calibration remains visiblynot_comparable; it is not silently treated as a stable trend metric.
If a judge determines that a detector signal is a false positive, a successful trace may be recorded as healthy while retaining the detector and judge evidence. A hard failed trace status remains a failure.
Budget and provider behavior
Reservations are created before provider egress and finalized from authoritative cost records. Cancellation and expiry release reserved spend. If a provider completion arrives after local cancellation, its actual cost is still charged and marked as a late completion. Unknown cost remains Unknown; it is never displayed as zero. A successful call with unavailable pricing still retains a cost-evidence ID marked
unknown_model, but that evidence anchor does not contribute a monetary amount to committed spend.
Provider failure behavior can continue deterministic checks, skip the provider check honestly, or fail closed according to the frozen policy. Operational health distinguishes provider_outage, budget_blocked, incomplete_evidence, and degraded from healthy.
Alerting and learning
Configure a failure-rate threshold, minimum severity, dedupe window, and one or more channels:- in-app notification;
- subscribed organization webhook;
- attributable Review Queue item.
Simulation and activation
Simulation is a persisted execution withrunType: simulation. It sorts historical traces deterministically, executes the exact frozen source predicate, applies the same sampling rules used by live traffic, and records:
- matched and sampled trace database IDs;
- exclusion counts by reason;
- estimated model calls and cost;
- expected alerts and historical failures;
- a bounded false-positive review set;
- budget preflight reasons and remaining limits;
- prompt, dataset, scorer, policy, and configuration hashes;
- missing evidence and warnings.
Run evidence and failure states
Every claimed sample creates a durable run before evaluation starts. Only one worker can claim a pending or retryable sample. A completed sample replay returns the persisted result instead of invoking the provider again. A run stores:- monitor version ID, number, configuration hash, and policy snapshot;
- sampled trace ID and key;
- exact scorer, prompt, and dataset snapshots;
- Model Gateway call ID;
- judge-result and cost-record IDs;
- reservation and actual cost;
- detector and judge decision summary;
- alert ID, missing evidence, attempt number, and retry history;
- start and completion timestamps.
Missing model call, cost, judge, prompt, dataset, scorer, or terminal result is listed explicitly. The UI never converts missing evidence into a successful verification claim.
Backfills
Backfills use an explicit immutable monitor version and a window of at most 31 days and 500 candidate traces per request. Simulation and preflight occur before jobs are enqueued. A backfill has its own execution, progress counters, sampled trace IDs, cost estimate, actual cost, retry history, and cancellation state. Alert delivery and candidate generation are disabled by default for backfills. Enable them explicitly when historical results should enter production triage. This prevents an exploratory backfill from flooding live alert channels. Cancellation marks pending samples as cancelled and releases active reservations. In-flight provider completions remain chargeable. Completed evidence is retained. Cancelled, failed, or partially completed backfills can be resumed with a new idempotency key; the resumed execution does not rewrite the original history.Alert triage and remediation
An alert records exact monitor, version, execution, sample, run, trace, model call, severity, category, dedupe key, delivery state, review item, candidate, and remediation state. Topic summaries are derived from persisted alert evidence and show count, open count, severity, and alert IDs. To resolve through remediation, provide:- a verification trace ID evaluated by the same monitor;
- the Experiment ID that proves the corrective comparison;
- the approved Prompt Hub version ID being promoted;
- a rationale and idempotency key.
Permissions
All services derive organization and actor from authenticated context. Body or query parameters cannot switch organization scope. Exact Prompt, Dataset, Scorer, Logs view, Experiment, trace, and alert references are rechecked against that organization.