> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Calibration control plane

> Version human anchors, approve score mappings, detect drift, preserve trend discontinuities, and explain release comparability gates.

# Calibration control plane

The Calibration control plane turns judge normalization into a reviewed, versioned evidence workflow. It separates four records that must not be conflated:

1. A stable **anchor-set identity** for one evaluation domain.
2. An immutable **anchor-set version** containing human scores and source provenance.
3. An immutable **mapping version** fit from one approved anchor version and one exact judge configuration.
4. Append-only **drift and release-gate decisions** that explain whether two score regimes are exact, calibrated, or not comparable.

Open **Calibration** at `/calibration`. The workspace is organization-scoped and shows approved anchors and mappings, blocking drift, release blocks, mapping confidence, and explicit trend-break rules.

<Warning>
  A legacy calibration run or score-mapping row is lineage evidence, not approval. Runtime normalization and new Scorer Studio bindings require the canonical reviewer-approved `mappingVersionId`.
</Warning>

## Permissions

| Action                                                             | Required authorization                        |
| ------------------------------------------------------------------ | --------------------------------------------- |
| Read the workspace, mapping detail, diffs, drift, and gate history | `eval:read`                                   |
| Create an anchor identity or new anchor version                    | `eval:write`                                  |
| Fit mappings, record drift, and evaluate gates                     | `runs:write`                                  |
| Approve or reject anchor and mapping versions                      | `scorer:publish` and organization admin       |
| Override a blocked comparability gate                              | Organization admin plus an explicit rationale |

API and database lookups include the active organization. A review, gate subject, judge configuration, scorer binding, anchor source, run, or mapping from another organization is rejected rather than disclosed.

## Build an anchor-set version

Choose **New anchor set** and define the domain, rubric version, score-scale version, and at least three human-scored anchors. Each anchor contains:

| Field                                 | Meaning                                         |
| ------------------------------------- | ----------------------------------------------- |
| `anchorKey`                           | Stable key unique within the version            |
| `input`                               | The exact case presented to the judge           |
| `expectedOutput` or `referenceAnswer` | Reviewed expected behavior                      |
| `humanScore`                          | Finite human score from 0 through 100           |
| `severityLabel` and `failureMode`     | Optional risk semantics                         |
| `sourceType`                          | `manual`, `review`, `dataset`, or `golden_case` |
| `sourceResourceId`                    | Durable link to the review, row, or golden case |
| `metadata`                            | Bounded structured source context               |

Creating the set writes both the stable identity and version 1. The version persists the complete anchors and a deterministic SHA-256 content hash. A separate append-only provenance row stores each anchor key, source link, input hash, expected-output hash, human score, severity, failure mode, and metadata.

Anchor content, provenance, creator, parent version, and hash cannot be updated or deleted. If labels or source evidence change, create a new version with the current version as its parent.

## Review anchor evidence

A draft cannot be used to fit a mapping. An admin reviewer chooses **Review**, inspects source links and human scores, and records an approval or rejection with a rationale of at least ten characters.

```text theme={null}
draft → approved
draft → rejected
approved → superseded by a newly approved version
```

Review records are append-only. Direct database approval without reviewer ID, timestamp, and rationale is rejected. Approving a new anchor version marks the previous approved version as superseded; it does not mutate or delete the old anchors.

## Fit a mapping version

On an approved anchor version, choose **Fit mapping**. Select the exact LLM judge configuration, mapping key, method, and one observation for every anchor.

```json theme={null}
{
  "mappingKey": "support_quality",
  "calibrationSetVersionId": "uuid",
  "judgeConfigId": 42,
  "method": "isotonic",
  "observations": [
    { "anchorKey": "critical_failure", "rawScore": 8, "parseOk": true },
    { "anchorKey": "decision_boundary", "rawScore": 54, "parseOk": true },
    { "anchorKey": "golden_success", "rawScore": 94, "parseOk": true }
  ]
}
```

Every anchor needs exactly one observation. Unknown keys, duplicate keys, missing observations, fewer than three successful parses, or anchors without a human score fail the fit.

### Supported methods

| Method     | Mapping behavior                                                          | Boundary behavior                                                                   |
| ---------- | ------------------------------------------------------------------------- | ----------------------------------------------------------------------------------- |
| Isotonic   | Pool-adjacent-violators monotonic fit from raw to human-normalized scores | Clamp below/above the learned range; interpolate between strictly increasing points |
| Percentile | Empirical cumulative rank across observed raw scores                      | Values below the minimum map to 0; values at/above the maximum map to 1             |
| Z-score    | `(raw - mean) / standardDeviation`                                        | Reject a non-finite or zero standard deviation                                      |

The draft mapping stores the exact anchor version, judge configuration, calibration run, legacy score mapping, method, score scale, function parameters, mapping hash, parent mapping version, and creator.

### Statistical evidence

The mapping registry exposes:

* Anchor count and parse-failure rate
* Raw-score mean, standard deviation, and variance
* Spearman correlation to human scores with average-rank tie handling
* Pass/fail agreement at the human decision boundary
* 95% mean interval and Fisher-transformed correlation interval when the sample supports it

Fit output is deterministic for the same canonical anchors and observations.

## Approve a mapping

Mapping review is independent from anchor review. The reviewer inspects method, confidence, parse failures, agreement, source version, parent diff, and drift state, then records an approval or rejection rationale.

A mapping cannot be approved when:

* Its anchor-set version is not approved.
* The judge configuration belongs to another organization.
* The run, score mapping, method, scale, or serialized mapping disagrees with the canonical lineage.
* Its drift state is `block`.
* Reviewer evidence is missing.

Approving a new version supersedes the previous approved mapping with the same mapping key. A superseded mapping retains reviewer approval for historical comparisons and exact-version scorer execution; it is no longer selected for new runtime resolutions.

## Compare mapping versions

Use `GET /api/calibration/mappings/{candidateId}/diff?against={baselineId}` to inspect:

* Method, score-scale, and mapping-hash changes
* Added, removed, and content-changed anchors
* Mean-score, agreement, and human-correlation deltas
* Parent/version lineage

The comparability result is:

| State            | Meaning                                                                                                       |
| ---------------- | ------------------------------------------------------------------------------------------------------------- |
| `exact`          | Same approved mapping version or identical immutable mapping hash                                             |
| `calibrated`     | Different approved/superseded mappings use the same anchor-set identity and score-scale version               |
| `not_comparable` | Missing evidence, draft/rejected mapping, blocking drift, different anchor identity, or different score scale |

`not_comparable` creates a structural trend discontinuity. The UI and release-gate explanation retain the reason; chart or report consumers must not draw a continuous calibrated trend through the boundary.

## Detect drift

Post a current observation for every anchor to `/api/calibration/mappings/{mappingVersionId}/drift`. EvalGate fits the observations with the mapping method and compares them with the immutable baseline statistics.

Default thresholds cover:

* Absolute raw mean shift
* Agreement-rate drop
* Human-correlation drop
* Parse-failure increase

The result is `stable`, `warning`, or `block`. Each event persists the observation hash, sample size, metric values, thresholds, reasons, actor, and timestamp. A metric between one and two times its threshold warns; a metric at least twice its threshold blocks. Drift events are append-only.

## Evaluate a release gate

A gate attaches to exactly one evaluation run or release artifact version. Supply optional baseline and candidate mapping-version IDs.

```json theme={null}
{
  "evaluationRunId": 1204,
  "baselineMappingVersionId": "baseline-uuid",
  "candidateMappingVersionId": "candidate-uuid"
}
```

The decision is `pass` for exact or calibrated evidence and `block` for not-comparable evidence. The stored explanation contains:

* Human-readable reasons
* Baseline and candidate mapping IDs and hashes
* Anchor-set version IDs
* Drift states
* The comparability state
* An explicit `trendDiscontinuity` flag

An admin can submit `override: true` with a rationale of at least 20 characters. The durable decision becomes `override`; the underlying state remains `not_comparable`. An override never relabels incompatible evidence as calibrated.

## Scorer Studio integration

Scorer Studio calibration accepts the canonical `mappingVersionId`. EvalGate derives and verifies the calibration set, run, and legacy score-mapping IDs from that version. Supplying mismatched legacy identifiers fails validation.

For a comparable scorer binding:

* The canonical mapping must be reviewer-approved with no blocking drift.
* The scorer binding must be explicitly approved.
* The binding is append-only and belongs to the exact immutable scorer version.
* Raw scorer output is converted from its declared score scale to the calibration raw 0–100 scale before the immutable mapping function is applied.
* Every scorer result persists the mapping version and matching lineage IDs.

The same mapping version may later become superseded. Existing scorer versions and historical results continue to reference and execute that exact version; new bindings resolve the newly approved mapping.

## Historical result retention

Judge results store the normalized score, mapping version, anchor-set identity/version, calibration run, score mapping, normalization method, score scale, comparability status, and group ID that existed at execution time.

Database guards reject updates to the raw score, judge configuration hash, creation time, normalized score, or calibration/comparability evidence. Recalibration creates a new mapping version and new results. It cannot silently rewrite a prior normalized score.

Scorer results are append-only and carry the canonical mapping-version reference whenever a normalized comparable score exists.

## API reference

| Method        | Endpoint                                                         | Purpose                                                    |
| ------------- | ---------------------------------------------------------------- | ---------------------------------------------------------- |
| `GET`         | `/api/calibration/workspace`                                     | List anchor versions, mappings, drift, and gate history    |
| `POST`        | `/api/calibration/sets`                                          | Create an anchor identity and draft version 1              |
| `POST`        | `/api/calibration/sets/{setId}/versions`                         | Create a lineage-linked anchor version                     |
| `POST`        | `/api/calibration/set-versions/{versionId}/review`               | Approve or reject anchor evidence                          |
| `POST`        | `/api/calibration/mappings`                                      | Fit a draft mapping version                                |
| `GET`         | `/api/calibration/mappings/{mappingVersionId}`                   | Read mapping, anchors, reviews, and drift                  |
| `POST`        | `/api/calibration/mappings/{mappingVersionId}/review`            | Approve or reject a mapping                                |
| `GET`         | `/api/calibration/mappings/{mappingVersionId}/diff?against={id}` | Diff two mapping versions                                  |
| `POST`        | `/api/calibration/mappings/{mappingVersionId}/drift`             | Persist a drift evaluation                                 |
| `GET`, `POST` | `/api/calibration/release-gates`                                 | Read or evaluate release comparability                     |
| `POST`        | `/api/scorers/{scorerId}/calibration`                            | Bind an exact approved mapping version to a scorer version |

## Failure behavior

| Condition                                                                        | Result                                               |
| -------------------------------------------------------------------------------- | ---------------------------------------------------- |
| Missing, duplicate, foreign, or unscored anchors                                 | `400 VALIDATION_ERROR` with the exact anchor problem |
| Anchor, judge, mapping, run, scorer, or gate subject is outside the organization | `404 NOT_FOUND` or database tenant-scope rejection   |
| Version changed or review already completed                                      | `409 CONFLICT`                                       |
| Mapping has blocking drift                                                       | `409 CONFLICT`                                       |
| Cross-version regimes are incompatible                                           | Persisted `block` with `not_comparable` reasons      |
| Non-admin attempts review or override                                            | `403 FORBIDDEN`                                      |
| Comparable scorer binding omits `mappingVersionId` or approval                   | `400 VALIDATION_ERROR`                               |
| A process attempts to mutate historical score evidence                           | Database operation rejected                          |

## Troubleshooting

* If an anchor review is blocked, inspect every source resource and create a new version instead of editing the draft content in place.
* If fitting fails, verify that every anchor key appears exactly once and at least three observations parsed successfully.
* If mapping approval is blocked, inspect drift, anchor approval, judge ownership, and the immutable run/mapping lineage.
* If a release is not comparable, read the stored reasons before considering an override; different anchor identities or score scales require a new shared calibration contract.
* If a Scorer Studio normalized score is withheld, bind the exact approved `mappingVersionId` to that scorer version. Do not copy legacy IDs from another version.
* If a chart shows a trend break, compare the mapping versions. Do not join raw or incompatible normalized scores across the boundary.

## Verification evidence

* Pure mapping, confidence, boundary, drift, diff, and invalid-cross-version tests: `tests/unit/calibration/contracts.test.ts`
* Runtime comparability contract tests: `tests/unit/judge/comparability-status.test.ts`
* Real PostgreSQL/PGlite lifecycle, provenance, approval, supersession, tenant isolation, historical retention, Scorer Studio binding, and gate override tests: `tests/lib/calibration-control-plane.db.test.ts`
* Authenticated API validation and permission tests: `tests/api/calibration.routes.test.ts`
* Workspace DOM state and authoring/reviewer tests: `tests/dom/calibration/calibration-control-plane.test.tsx`
* Authenticated release-gate golden path: `e2e/calibration-release-gate-golden-path.spec.ts`