Experiments

Experiments turn a mutable Playground comparison into a named, content-addressed evaluation record. Use them when a model, prompt, tool, scorer, routing policy, or dataset change needs reproducible evidence before release.

An Experiment never follows a mutable alias. It freezes the exact inputs available at creation time and executes those values again on every rerun.

What an Experiment freezes

The snapshot is immutable and protected by database triggers. It includes:

every evaluation test-case input, expected output, metadata value, lifecycle field, and source reference;
the selected Dataset Hub version, schema, content hash, row order, row payloads, and row content hashes when a dataset version is attached;
baseline and candidate prompt version identifiers, prompt content hashes, full messages, and system prompts;
provider, model, routing profile and strategy, parameters, seed, and environment;
tool configuration and policy snapshots;
complete scorer definitions, scorer prompt hashes, and settings;
calibration mapping identifiers, score scales, methods, and mapping payloads;
evaluation execution settings, model settings, custom metrics, executor configuration, runner version, code version, and snapshot schema version.

The experiment record stores a SHA-256 hash of the canonical snapshot. Each variant has its own configuration hash. Changing a live prompt, Playground variant, dataset row, scorer, or calibration mapping after creation does not rewrite either value.

Recommended workflow

Prepare the matrix in Playground

Create one baseline and at least one candidate. Select the exact provider, model, tools, parameters, scorers, and policy configuration that you intend to compare.

Publish reusable inputs

Publish the Dataset Hub version and approve any referenced Prompt Hub version. Draft or unapproved versions are rejected when the experiment is created.

Promote to an Experiment

Choose Promote from Playground, name the experiment, and record the expected dataset and variant hashes. If a live asset changes before promotion completes, creation fails with a stale_asset conflict instead of silently freezing a different value.

Run the matrix

Select Run matrix. Every variant receives a normal evaluation run, and every row receives a normal test result. Experiment bindings preserve the exact execution, variant, run, and test-result lineage.

Inspect the comparison

Review paired rows, regressions, score distributions, uncertainty, cost, p50/p95 latency, failures, tool errors, policy blocks, unknown values, and evidence completeness together.

Record a decision

Select a candidate and write a rationale. Winner approval is blocked when the candidate violates the frozen winner policy, introduces disallowed regressions, has incomplete cost, or crosses an uncalibrated judge boundary.

Create a publish proposal

An accepted winner can create a proposal for a prompt version, Evals-as-Code manifest, online monitor, release artifact, signed report, or release gate. The proposal is always pending_approval; the Experiment does not make a live change.

Comparison semantics

Each candidate is paired to the baseline by frozen test-case identifier. The comparison reports:

Dimension	Evidence shown
Quality	Baseline and candidate pass rates, delta, scorer distributions, score mean delta, and paired-row changes
Uncertainty	A 95% interval for the pass-rate difference when enough observations exist
Cost	Total known cost, delta, and separate unknown-cost row counts
Latency	Baseline and candidate p50/p95 plus both deltas
Failures	Failure-mode counts, new regressions, fixed regressions, tool-error delta, and policy-block delta
Provenance	Evaluation run, test result, model call, trace, evidence completeness, prompt/scorer hashes, and comparison hash

Every candidate receives one outcome:

improvement: pass rate or paired outcomes improve without a new paired regression;
regression: quality falls or the candidate introduces a paired regression;
tradeoff: the candidate fixes some rows and regresses others;
tie: complete paired quality is unchanged;
inconclusive: required paired data is missing or uncertainty crosses zero;
not_comparable: judge or calibration identities differ without a shared scale.

The Summary tab shows the decision logic used for the classification. Missing data and judge discontinuities appear as warnings; neither is converted to zero.

Experiment workspace

The detail workspace contains eight sections:

Summary combines headline quality, cost, latency, classification, warnings, and decision logic.
Variants shows the baseline and candidates with model, provider, scorer, calibration, and configuration hashes.
Rows is a virtualized paired matrix with baseline output, candidate output, scorer delta, cost, latency, and classification.
Regressions focuses on newly failing and newly fixed cases.
Metrics shows distributions, uncertainty, operational tradeoffs, tool failures, and policy blocks.
Traces links normal run, test-result, model-call, trace, and evidence identifiers.
Decision records the candidate, comparison hash, rationale, actor, and decision time.
Evidence shows publish proposals and the full frozen snapshot payload.

Row filters include regression, improvement, disagreement, unchanged, missing, high cost, tool error, policy block, and free-text category or tag search. The matrix only renders visible rows, so large datasets remain responsive without changing the comparison result.

Progress, cancellation, and reruns

Execution progress reports completed variants and processed rows. A cancellation request stops the matrix before the next variant. Completed variant bindings remain append-only evidence. Use Rerun snapshot to execute the same immutable values again. The new execution records its parent execution and increments the attempt number. Only one queued, running, or cancellation-pending execution can exist for an Experiment at a time.

Rerunning an Experiment does not refresh prompt aliases, dataset versions, scorer settings, calibration mappings, or policies. Create a new Experiment to test newer inputs.

Empty and blocked states

State	Meaning	Next action
`not_run`	No matrix execution exists	Run the frozen matrix
`partial`	A variant, paired row, cost, latency, or evidence link is missing	Inspect failed runs and rerun the same snapshot
`incomparable`	Judge or calibration identities do not share a scale	Create a calibrated mapping or a new comparable Experiment
`budget_blocked`	Estimated matrix cost exceeds the frozen budget policy	Reduce the matrix or create a new snapshot with an authorized budget
`stale_asset`	An expected hash changed before snapshot creation or run submission	Refresh the asset and explicitly create a new snapshot
No winner	No eligible accepted candidate exists	Record `needs_review` or `reject`; do not publish

API reference

All endpoints are organization-scoped. UUIDs from another organization or another Experiment are rejected by composite database constraints, even when the individual identifier exists.

Method	Endpoint	Purpose	Required scope
`GET`	`/api/experiments`	List Experiments with optional evaluation and status filters	`eval:read`
`POST`	`/api/experiments`	Create an immutable API-configured snapshot	`eval:write`
`GET`	`/api/experiments/:id`	Read snapshot, variants, executions, decisions, and proposals	`eval:read`
`POST`	`/api/experiments/:id/run`	Execute or rerun the frozen matrix	`runs:write`
`POST`	`/api/experiments/:id/cancel`	Request cancellation of the active execution	`runs:write`
`GET`	`/api/experiments/:id/progress`	Read variant and row progress	`runs:read`
`GET`	`/api/experiments/:id/results`	Read normal run/test-result evidence with provenance	`runs:read`
`GET`	`/api/experiments/:id/compare`	Read paired comparisons and workspace state	`runs:read`
`POST`	`/api/experiments/:id/decision`	Record accept, reject, or needs-review evidence	`eval:write` and admin role
`POST`	`/api/experiments/:id/publish-winner`	Create an exact-snapshot publish proposal	`prompt:publish` and admin role
`POST`	`/api/evaluations/:id/playground/promote-to-experiment`	Freeze the current Playground matrix	`eval:write`

Idempotency

Creation, run, decision, and publish requests require an idempotency key. Replaying the same key and same request returns the existing record. Reusing a key with a different snapshot, rerun parent, winner, or target returns 409 CONFLICT.

Create an Experiment

{
  "name": "Checkout prompt v8",
  "description": "Candidate compared against the production prompt",
  "evaluationId": 42,
  "datasetVersionId": "c55e41d2-e234-45aa-8dd9-2ea901ad85ff",
  "expectedDatasetContentHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "baselineVariantKey": "production",
  "variants": [
    {
      "key": "production",
      "name": "Production",
      "promptVersionId": "101",
      "expectedPromptContentHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "scorerConfigIds": [12],
      "calibrationMappingIds": ["mapping-2026-07"]
    },
    {
      "key": "candidate",
      "name": "Concise checkout",
      "promptVersionId": "108",
      "expectedPromptContentHash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "parameters": { "temperature": 0 }
    }
  ],
  "environment": "staging",
  "seed": 17,
  "runnerVersion": "evalgate-runner-3.7.0",
  "codeVersion": "8c17162",
  "policies": { "budget": { "maxCostUsd": 25 } },
  "winnerPolicy": {
    "minimumPassRateDelta": 0,
    "maximumCostIncreasePercent": 15,
    "maximumP95LatencyIncreasePercent": 10,
    "allowIncomparableJudges": false,
    "allowRegressions": false
  },
  "idempotencyKey": "checkout-prompt-v8-create"
}

Run and rerun

{
  "idempotencyKey": "checkout-prompt-v8-attempt-2",
  "parentExecutionId": "8c8151eb-39dc-43ae-9b74-c7b21ee75221",
  "expectedSnapshotHash": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
  "estimatedCostUsd": 12.4
}

Omit parentExecutionId for the first execution. A rerun parent must be terminal and belong to the same organization and Experiment.

Approve a winner

{
  "decision": "accept",
  "winningVariantKey": "candidate",
  "rationale": "Pass rate improved by 4.2 points with no paired regressions; cost rose 3.1% and p95 fell 8%.",
  "comparisonHash": "the exact comparison hash shown in the workspace",
  "idempotencyKey": "checkout-v8-decision"
}

An accepted decision requires an eligible non-baseline candidate. reject and needs_review decisions must not provide a winning variant.

Propose publication

{
  "decisionId": "e80a6b88-af07-4e70-a3cf-20d93ea11a2a",
  "target": "release_gate",
  "environment": "staging",
  "idempotencyKey": "checkout-v8-release-gate-proposal"
}

The response contains status: "pending_approval", apply: null, the Experiment snapshot hash, winner configuration hash, comparison hash, and decision identifier.

Audit evidence

The audit log records snapshot creation, execution completion or cancellation, winner decisions, and publish proposals. Use the following identifiers when exporting evidence:

Experiment ID and snapshot hash;
Dataset version and content hash;
variant ID and configuration hash;
execution ID, attempt, and rerun parent;
evaluation run and test-result IDs;
model-call, trace, and evidence identifiers;
decision ID, actor, rationale, and comparison hash;
publication proposal ID, target, environment, and pending_approval status.

These links allow an auditor to move from the release proposal back to the exact row output and frozen input that justified it.

Get Started

Core Concepts

Guides

SDK Reference

Platform

What an Experiment freezes

Recommended workflow

Comparison semantics

Experiment workspace

Progress, cancellation, and reruns

Empty and blocked states

API reference

Idempotency

Create an Experiment

Run and rerun

Approve a winner

Propose publication

Audit evidence

​What an Experiment freezes

​Recommended workflow

​Comparison semantics

​Experiment workspace

​Progress, cancellation, and reruns

​Empty and blocked states

​API reference

​Idempotency

​Create an Experiment

​Run and rerun

​Approve a winner

​Propose publication

​Audit evidence

What an Experiment freezes

Recommended workflow

Comparison semantics

Experiment workspace

Progress, cancellation, and reruns

Empty and blocked states

API reference

Idempotency

Create an Experiment

Run and rerun

Approve a winner

Propose publication

Audit evidence