Skip to main content

​
Calibrate a scorer

Calibration makes a scorer’s scores comparable across versions and models. Without it, a “0.8” from one judge does not mean the same thing as a “0.8” from another. Calibration lives at /calibration.

​
Step 1 — Build an anchor set

An anchor set is a small, stable set of labeled cases that represent the range of quality you care about. Create one, label every item, and version it. A screenshot of the calibration workspace with an anchor set listed.

​
Step 2 — Run the scorer on the anchors

Run the current scorer version against the anchor set. EvalGate records the raw scores and the labels.

​
Step 3 — Review the mapping

The mapping converts raw scores to a normalized scale (for example, 0–100) so different scorers can be compared. Review the mapping, then submit it for review. A screenshot of the mapping review showing raw-to-normalized score curves.

​
Step 4 — Check drift

When you publish a new scorer version, compare its mapping against the previous one. If the curves drifted, the new version is not comparable to the old — re-baseline before using it as a release gate.

​
Step 5 — Bind to the release gate

Bind the calibrated scorer version to an evaluation’s release gate. Now a release can only pass if the new scorer beats the calibrated baseline on the anchor set.

​
Up next

Try a change safely in the Playground.