> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Run immutable baseline-versus-candidate matrices, inspect paired evidence, approve an attributable winner, and create a governed publish proposal.

Experiments turn a mutable Playground comparison into a named, content-addressed evaluation record. Use them when a model, prompt, tool, scorer, routing policy, or dataset change needs reproducible evidence before release.

<Info>
  An Experiment never follows a mutable alias. It freezes the exact inputs available at creation time and executes those values again on every rerun.
</Info>

## What an Experiment freezes

The snapshot is immutable and protected by database triggers. It includes:

* every evaluation test-case input, expected output, metadata value, lifecycle field, and source reference;
* the selected Dataset Hub version, schema, content hash, row order, row payloads, and row content hashes when a dataset version is attached;
* baseline and candidate prompt version identifiers, prompt content hashes, full messages, and system prompts;
* provider, model, routing profile and strategy, parameters, seed, and environment;
* tool configuration and policy snapshots;
* complete scorer definitions, scorer prompt hashes, and settings;
* calibration mapping identifiers, score scales, methods, and mapping payloads;
* evaluation execution settings, model settings, custom metrics, executor configuration, runner version, code version, and snapshot schema version.

The experiment record stores a SHA-256 hash of the canonical snapshot. Each variant has its own configuration hash. Changing a live prompt, Playground variant, dataset row, scorer, or calibration mapping after creation does not rewrite either value.

## Recommended workflow

<Steps>
  <Step title="Prepare the matrix in Playground">
    Create one baseline and at least one candidate. Select the exact provider, model, tools, parameters, scorers, and policy configuration that you intend to compare.
  </Step>

  <Step title="Publish reusable inputs">
    Publish the Dataset Hub version and approve any referenced Prompt Hub version. Draft or unapproved versions are rejected when the experiment is created.
  </Step>

  <Step title="Promote to an Experiment">
    Choose **Promote from Playground**, name the experiment, and record the expected dataset and variant hashes. If a live asset changes before promotion completes, creation fails with a `stale_asset` conflict instead of silently freezing a different value.
  </Step>

  <Step title="Run the matrix">
    Select **Run matrix**. Every variant receives a normal evaluation run, and every row receives a normal test result. Experiment bindings preserve the exact execution, variant, run, and test-result lineage.
  </Step>

  <Step title="Inspect the comparison">
    Review paired rows, regressions, score distributions, uncertainty, cost, p50/p95 latency, failures, tool errors, policy blocks, unknown values, and evidence completeness together.
  </Step>

  <Step title="Record a decision">
    Select a candidate and write a rationale. Winner approval is blocked when the candidate violates the frozen winner policy, introduces disallowed regressions, has incomplete cost, or crosses an uncalibrated judge boundary.
  </Step>

  <Step title="Create a publish proposal">
    An accepted winner can create a proposal for a prompt version, Evals-as-Code manifest, online monitor, release artifact, signed report, or release gate. The proposal is always `pending_approval`; the Experiment does not make a live change.
  </Step>
</Steps>

## Comparison semantics

Each candidate is paired to the baseline by frozen test-case identifier. The comparison reports:

| Dimension   | Evidence shown                                                                                                   |
| ----------- | ---------------------------------------------------------------------------------------------------------------- |
| Quality     | Baseline and candidate pass rates, delta, scorer distributions, score mean delta, and paired-row changes         |
| Uncertainty | A 95% interval for the pass-rate difference when enough observations exist                                       |
| Cost        | Total known cost, delta, and separate unknown-cost row counts                                                    |
| Latency     | Baseline and candidate p50/p95 plus both deltas                                                                  |
| Failures    | Failure-mode counts, new regressions, fixed regressions, tool-error delta, and policy-block delta                |
| Provenance  | Evaluation run, test result, model call, trace, evidence completeness, prompt/scorer hashes, and comparison hash |

Every candidate receives one outcome:

* `improvement`: pass rate or paired outcomes improve without a new paired regression;
* `regression`: quality falls or the candidate introduces a paired regression;
* `tradeoff`: the candidate fixes some rows and regresses others;
* `tie`: complete paired quality is unchanged;
* `inconclusive`: required paired data is missing or uncertainty crosses zero;
* `not_comparable`: judge or calibration identities differ without a shared scale.

The **Summary** tab shows the decision logic used for the classification. Missing data and judge discontinuities appear as warnings; neither is converted to zero.

## Experiment workspace

The detail workspace contains eight sections:

* **Summary** combines headline quality, cost, latency, classification, warnings, and decision logic.
* **Variants** shows the baseline and candidates with model, provider, scorer, calibration, and configuration hashes.
* **Rows** is a virtualized paired matrix with baseline output, candidate output, scorer delta, cost, latency, and classification.
* **Regressions** focuses on newly failing and newly fixed cases.
* **Metrics** shows distributions, uncertainty, operational tradeoffs, tool failures, and policy blocks.
* **Traces** links normal run, test-result, model-call, trace, and evidence identifiers.
* **Decision** records the candidate, comparison hash, rationale, actor, and decision time.
* **Evidence** shows publish proposals and the full frozen snapshot payload.

Row filters include regression, improvement, disagreement, unchanged, missing, high cost, tool error, policy block, and free-text category or tag search. The matrix only renders visible rows, so large datasets remain responsive without changing the comparison result.

## Progress, cancellation, and reruns

Execution progress reports completed variants and processed rows. A cancellation request stops the matrix before the next variant. Completed variant bindings remain append-only evidence.

Use **Rerun snapshot** to execute the same immutable values again. The new execution records its parent execution and increments the attempt number. Only one queued, running, or cancellation-pending execution can exist for an Experiment at a time.

<Warning>
  Rerunning an Experiment does not refresh prompt aliases, dataset versions, scorer settings, calibration mappings, or policies. Create a new Experiment to test newer inputs.
</Warning>

## Empty and blocked states

| State            | Meaning                                                             | Next action                                                          |
| ---------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------- |
| `not_run`        | No matrix execution exists                                          | Run the frozen matrix                                                |
| `partial`        | A variant, paired row, cost, latency, or evidence link is missing   | Inspect failed runs and rerun the same snapshot                      |
| `incomparable`   | Judge or calibration identities do not share a scale                | Create a calibrated mapping or a new comparable Experiment           |
| `budget_blocked` | Estimated matrix cost exceeds the frozen budget policy              | Reduce the matrix or create a new snapshot with an authorized budget |
| `stale_asset`    | An expected hash changed before snapshot creation or run submission | Refresh the asset and explicitly create a new snapshot               |
| No winner        | No eligible accepted candidate exists                               | Record `needs_review` or `reject`; do not publish                    |

## API reference

All endpoints are organization-scoped. UUIDs from another organization or another Experiment are rejected by composite database constraints, even when the individual identifier exists.

| Method | Endpoint                                                | Purpose                                                       | Required scope                  |
| ------ | ------------------------------------------------------- | ------------------------------------------------------------- | ------------------------------- |
| `GET`  | `/api/experiments`                                      | List Experiments with optional evaluation and status filters  | `eval:read`                     |
| `POST` | `/api/experiments`                                      | Create an immutable API-configured snapshot                   | `eval:write`                    |
| `GET`  | `/api/experiments/:id`                                  | Read snapshot, variants, executions, decisions, and proposals | `eval:read`                     |
| `POST` | `/api/experiments/:id/run`                              | Execute or rerun the frozen matrix                            | `runs:write`                    |
| `POST` | `/api/experiments/:id/cancel`                           | Request cancellation of the active execution                  | `runs:write`                    |
| `GET`  | `/api/experiments/:id/progress`                         | Read variant and row progress                                 | `runs:read`                     |
| `GET`  | `/api/experiments/:id/results`                          | Read normal run/test-result evidence with provenance          | `runs:read`                     |
| `GET`  | `/api/experiments/:id/compare`                          | Read paired comparisons and workspace state                   | `runs:read`                     |
| `POST` | `/api/experiments/:id/decision`                         | Record accept, reject, or needs-review evidence               | `eval:write` and admin role     |
| `POST` | `/api/experiments/:id/publish-winner`                   | Create an exact-snapshot publish proposal                     | `prompt:publish` and admin role |
| `POST` | `/api/evaluations/:id/playground/promote-to-experiment` | Freeze the current Playground matrix                          | `eval:write`                    |

### Idempotency

Creation, run, decision, and publish requests require an idempotency key. Replaying the same key and same request returns the existing record. Reusing a key with a different snapshot, rerun parent, winner, or target returns `409 CONFLICT`.

### Create an Experiment

```json theme={null}
{
  "name": "Checkout prompt v8",
  "description": "Candidate compared against the production prompt",
  "evaluationId": 42,
  "datasetVersionId": "c55e41d2-e234-45aa-8dd9-2ea901ad85ff",
  "expectedDatasetContentHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "baselineVariantKey": "production",
  "variants": [
    {
      "key": "production",
      "name": "Production",
      "promptVersionId": "101",
      "expectedPromptContentHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "scorerConfigIds": [12],
      "calibrationMappingIds": ["mapping-2026-07"]
    },
    {
      "key": "candidate",
      "name": "Concise checkout",
      "promptVersionId": "108",
      "expectedPromptContentHash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "parameters": { "temperature": 0 }
    }
  ],
  "environment": "staging",
  "seed": 17,
  "runnerVersion": "evalgate-runner-3.7.0",
  "codeVersion": "8c17162",
  "policies": { "budget": { "maxCostUsd": 25 } },
  "winnerPolicy": {
    "minimumPassRateDelta": 0,
    "maximumCostIncreasePercent": 15,
    "maximumP95LatencyIncreasePercent": 10,
    "allowIncomparableJudges": false,
    "allowRegressions": false
  },
  "idempotencyKey": "checkout-prompt-v8-create"
}
```

### Run and rerun

```json theme={null}
{
  "idempotencyKey": "checkout-prompt-v8-attempt-2",
  "parentExecutionId": "8c8151eb-39dc-43ae-9b74-c7b21ee75221",
  "expectedSnapshotHash": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
  "estimatedCostUsd": 12.4
}
```

Omit `parentExecutionId` for the first execution. A rerun parent must be terminal and belong to the same organization and Experiment.

### Approve a winner

```json theme={null}
{
  "decision": "accept",
  "winningVariantKey": "candidate",
  "rationale": "Pass rate improved by 4.2 points with no paired regressions; cost rose 3.1% and p95 fell 8%.",
  "comparisonHash": "the exact comparison hash shown in the workspace",
  "idempotencyKey": "checkout-v8-decision"
}
```

An accepted decision requires an eligible non-baseline candidate. `reject` and `needs_review` decisions must not provide a winning variant.

### Propose publication

```json theme={null}
{
  "decisionId": "e80a6b88-af07-4e70-a3cf-20d93ea11a2a",
  "target": "release_gate",
  "environment": "staging",
  "idempotencyKey": "checkout-v8-release-gate-proposal"
}
```

The response contains `status: "pending_approval"`, `apply: null`, the Experiment snapshot hash, winner configuration hash, comparison hash, and decision identifier.

## Audit evidence

The audit log records snapshot creation, execution completion or cancellation, winner decisions, and publish proposals. Use the following identifiers when exporting evidence:

* Experiment ID and snapshot hash;
* Dataset version and content hash;
* variant ID and configuration hash;
* execution ID, attempt, and rerun parent;
* evaluation run and test-result IDs;
* model-call, trace, and evidence identifiers;
* decision ID, actor, rationale, and comparison hash;
* publication proposal ID, target, environment, and `pending_approval` status.

These links allow an auditor to move from the release proposal back to the exact row output and frozen input that justified it.