Calibration control plane

The Calibration control plane turns judge normalization into a reviewed, versioned evidence workflow. It separates four records that must not be conflated:

A stable anchor-set identity for one evaluation domain.
An immutable anchor-set version containing human scores and source provenance.
An immutable mapping version fit from one approved anchor version and one exact judge configuration.
Append-only drift and release-gate decisions that explain whether two score regimes are exact, calibrated, or not comparable.

Open Calibration at /calibration. The workspace is organization-scoped and shows approved anchors and mappings, blocking drift, release blocks, mapping confidence, and explicit trend-break rules.

A legacy calibration run or score-mapping row is lineage evidence, not approval. Runtime normalization and new Scorer Studio bindings require the canonical reviewer-approved mappingVersionId.

Permissions

Action	Required authorization
Read the workspace, mapping detail, diffs, drift, and gate history	`eval:read`
Create an anchor identity or new anchor version	`eval:write`
Fit mappings, record drift, and evaluate gates	`runs:write`
Approve or reject anchor and mapping versions	`scorer:publish` and organization admin
Override a blocked comparability gate	Organization admin plus an explicit rationale

API and database lookups include the active organization. A review, gate subject, judge configuration, scorer binding, anchor source, run, or mapping from another organization is rejected rather than disclosed.

Build an anchor-set version

Choose New anchor set and define the domain, rubric version, score-scale version, and at least three human-scored anchors. Each anchor contains:

Field	Meaning
`anchorKey`	Stable key unique within the version
`input`	The exact case presented to the judge
`expectedOutput` or `referenceAnswer`	Reviewed expected behavior
`humanScore`	Finite human score from 0 through 100
`severityLabel` and `failureMode`	Optional risk semantics
`sourceType`	`manual`, `review`, `dataset`, or `golden_case`
`sourceResourceId`	Durable link to the review, row, or golden case
`metadata`	Bounded structured source context

Creating the set writes both the stable identity and version 1. The version persists the complete anchors and a deterministic SHA-256 content hash. A separate append-only provenance row stores each anchor key, source link, input hash, expected-output hash, human score, severity, failure mode, and metadata. Anchor content, provenance, creator, parent version, and hash cannot be updated or deleted. If labels or source evidence change, create a new version with the current version as its parent.

Review anchor evidence

A draft cannot be used to fit a mapping. An admin reviewer chooses Review, inspects source links and human scores, and records an approval or rejection with a rationale of at least ten characters.

draft → approved
draft → rejected
approved → superseded by a newly approved version

Review records are append-only. Direct database approval without reviewer ID, timestamp, and rationale is rejected. Approving a new anchor version marks the previous approved version as superseded; it does not mutate or delete the old anchors.

Fit a mapping version

On an approved anchor version, choose Fit mapping. Select the exact LLM judge configuration, mapping key, method, and one observation for every anchor.

{
  "mappingKey": "support_quality",
  "calibrationSetVersionId": "uuid",
  "judgeConfigId": 42,
  "method": "isotonic",
  "observations": [
    { "anchorKey": "critical_failure", "rawScore": 8, "parseOk": true },
    { "anchorKey": "decision_boundary", "rawScore": 54, "parseOk": true },
    { "anchorKey": "golden_success", "rawScore": 94, "parseOk": true }
  ]
}

Every anchor needs exactly one observation. Unknown keys, duplicate keys, missing observations, fewer than three successful parses, or anchors without a human score fail the fit.

Supported methods

Method	Mapping behavior	Boundary behavior
Isotonic	Pool-adjacent-violators monotonic fit from raw to human-normalized scores	Clamp below/above the learned range; interpolate between strictly increasing points
Percentile	Empirical cumulative rank across observed raw scores	Values below the minimum map to 0; values at/above the maximum map to 1
Z-score	`(raw - mean) / standardDeviation`	Reject a non-finite or zero standard deviation

The draft mapping stores the exact anchor version, judge configuration, calibration run, legacy score mapping, method, score scale, function parameters, mapping hash, parent mapping version, and creator.

Statistical evidence

The mapping registry exposes:

Anchor count and parse-failure rate
Raw-score mean, standard deviation, and variance
Spearman correlation to human scores with average-rank tie handling
Pass/fail agreement at the human decision boundary
95% mean interval and Fisher-transformed correlation interval when the sample supports it

Fit output is deterministic for the same canonical anchors and observations.

Approve a mapping

Mapping review is independent from anchor review. The reviewer inspects method, confidence, parse failures, agreement, source version, parent diff, and drift state, then records an approval or rejection rationale. A mapping cannot be approved when:

Its anchor-set version is not approved.
The judge configuration belongs to another organization.
The run, score mapping, method, scale, or serialized mapping disagrees with the canonical lineage.
Its drift state is block.
Reviewer evidence is missing.

Approving a new version supersedes the previous approved mapping with the same mapping key. A superseded mapping retains reviewer approval for historical comparisons and exact-version scorer execution; it is no longer selected for new runtime resolutions.

Compare mapping versions

Use GET /api/calibration/mappings/{candidateId}/diff?against={baselineId} to inspect:

Method, score-scale, and mapping-hash changes
Added, removed, and content-changed anchors
Mean-score, agreement, and human-correlation deltas
Parent/version lineage

The comparability result is:

State	Meaning
`exact`	Same approved mapping version or identical immutable mapping hash
`calibrated`	Different approved/superseded mappings use the same anchor-set identity and score-scale version
`not_comparable`	Missing evidence, draft/rejected mapping, blocking drift, different anchor identity, or different score scale

not_comparable creates a structural trend discontinuity. The UI and release-gate explanation retain the reason; chart or report consumers must not draw a continuous calibrated trend through the boundary.

Detect drift

Post a current observation for every anchor to /api/calibration/mappings/{mappingVersionId}/drift. EvalGate fits the observations with the mapping method and compares them with the immutable baseline statistics. Default thresholds cover:

Absolute raw mean shift
Agreement-rate drop
Human-correlation drop
Parse-failure increase

The result is stable, warning, or block. Each event persists the observation hash, sample size, metric values, thresholds, reasons, actor, and timestamp. A metric between one and two times its threshold warns; a metric at least twice its threshold blocks. Drift events are append-only.

Evaluate a release gate

A gate attaches to exactly one evaluation run or release artifact version. Supply optional baseline and candidate mapping-version IDs.

{
  "evaluationRunId": 1204,
  "baselineMappingVersionId": "baseline-uuid",
  "candidateMappingVersionId": "candidate-uuid"
}

The decision is pass for exact or calibrated evidence and block for not-comparable evidence. The stored explanation contains:

Human-readable reasons
Baseline and candidate mapping IDs and hashes
Anchor-set version IDs
Drift states
The comparability state
An explicit trendDiscontinuity flag

An admin can submit override: true with a rationale of at least 20 characters. The durable decision becomes override; the underlying state remains not_comparable. An override never relabels incompatible evidence as calibrated.

Scorer Studio integration

Scorer Studio calibration accepts the canonical mappingVersionId. EvalGate derives and verifies the calibration set, run, and legacy score-mapping IDs from that version. Supplying mismatched legacy identifiers fails validation. For a comparable scorer binding:

The canonical mapping must be reviewer-approved with no blocking drift.
The scorer binding must be explicitly approved.
The binding is append-only and belongs to the exact immutable scorer version.
Raw scorer output is converted from its declared score scale to the calibration raw 0–100 scale before the immutable mapping function is applied.
Every scorer result persists the mapping version and matching lineage IDs.

The same mapping version may later become superseded. Existing scorer versions and historical results continue to reference and execute that exact version; new bindings resolve the newly approved mapping.

Historical result retention

Judge results store the normalized score, mapping version, anchor-set identity/version, calibration run, score mapping, normalization method, score scale, comparability status, and group ID that existed at execution time. Database guards reject updates to the raw score, judge configuration hash, creation time, normalized score, or calibration/comparability evidence. Recalibration creates a new mapping version and new results. It cannot silently rewrite a prior normalized score. Scorer results are append-only and carry the canonical mapping-version reference whenever a normalized comparable score exists.

API reference

Method	Endpoint	Purpose
`GET`	`/api/calibration/workspace`	List anchor versions, mappings, drift, and gate history
`POST`	`/api/calibration/sets`	Create an anchor identity and draft version 1
`POST`	`/api/calibration/sets/{setId}/versions`	Create a lineage-linked anchor version
`POST`	`/api/calibration/set-versions/{versionId}/review`	Approve or reject anchor evidence
`POST`	`/api/calibration/mappings`	Fit a draft mapping version
`GET`	`/api/calibration/mappings/{mappingVersionId}`	Read mapping, anchors, reviews, and drift
`POST`	`/api/calibration/mappings/{mappingVersionId}/review`	Approve or reject a mapping
`GET`	`/api/calibration/mappings/{mappingVersionId}/diff?against={id}`	Diff two mapping versions
`POST`	`/api/calibration/mappings/{mappingVersionId}/drift`	Persist a drift evaluation
`GET`, `POST`	`/api/calibration/release-gates`	Read or evaluate release comparability
`POST`	`/api/scorers/{scorerId}/calibration`	Bind an exact approved mapping version to a scorer version

Failure behavior

Condition	Result
Missing, duplicate, foreign, or unscored anchors	`400 VALIDATION_ERROR` with the exact anchor problem
Anchor, judge, mapping, run, scorer, or gate subject is outside the organization	`404 NOT_FOUND` or database tenant-scope rejection
Version changed or review already completed	`409 CONFLICT`
Mapping has blocking drift	`409 CONFLICT`
Cross-version regimes are incompatible	Persisted `block` with `not_comparable` reasons
Non-admin attempts review or override	`403 FORBIDDEN`
Comparable scorer binding omits `mappingVersionId` or approval	`400 VALIDATION_ERROR`
A process attempts to mutate historical score evidence	Database operation rejected

Troubleshooting

If an anchor review is blocked, inspect every source resource and create a new version instead of editing the draft content in place.
If fitting fails, verify that every anchor key appears exactly once and at least three observations parsed successfully.
If mapping approval is blocked, inspect drift, anchor approval, judge ownership, and the immutable run/mapping lineage.
If a release is not comparable, read the stored reasons before considering an override; different anchor identities or score scales require a new shared calibration contract.
If a Scorer Studio normalized score is withheld, bind the exact approved mappingVersionId to that scorer version. Do not copy legacy IDs from another version.
If a chart shows a trend break, compare the mapping versions. Do not join raw or incompatible normalized scores across the boundary.

Verification evidence

Pure mapping, confidence, boundary, drift, diff, and invalid-cross-version tests: tests/unit/calibration/contracts.test.ts
Runtime comparability contract tests: tests/unit/judge/comparability-status.test.ts
Real PostgreSQL/PGlite lifecycle, provenance, approval, supersession, tenant isolation, historical retention, Scorer Studio binding, and gate override tests: tests/lib/calibration-control-plane.db.test.ts
Authenticated API validation and permission tests: tests/api/calibration.routes.test.ts
Workspace DOM state and authoring/reviewer tests: tests/dom/calibration/calibration-control-plane.test.tsx
Authenticated release-gate golden path: e2e/calibration-release-gate-golden-path.spec.ts

Get Started

Core Concepts

Guides

SDK Reference

Platform

Calibration control plane

Calibration control plane

Permissions

Build an anchor-set version

Review anchor evidence

Fit a mapping version

Supported methods

Statistical evidence

Approve a mapping

Compare mapping versions

Detect drift

Evaluate a release gate

Scorer Studio integration

Historical result retention

API reference

Failure behavior

Troubleshooting

Verification evidence

​Calibration control plane

​Permissions

​Build an anchor-set version

​Review anchor evidence

​Fit a mapping version

​Supported methods

​Statistical evidence

​Approve a mapping

​Compare mapping versions

​Detect drift

​Evaluate a release gate

​Scorer Studio integration

​Historical result retention

​API reference

​Failure behavior

​Troubleshooting

​Verification evidence

Calibration control plane

Permissions

Build an anchor-set version

Review anchor evidence

Fit a mapping version

Supported methods

Statistical evidence

Approve a mapping

Compare mapping versions

Detect drift

Evaluate a release gate

Scorer Studio integration

Historical result retention

API reference

Failure behavior

Troubleshooting

Verification evidence