---
title: "How to Calibrate an LLM Judge Before Trusting Its Scores"
description: "Measure false passes, false failures, stability, and drift against a reviewed human anchor set."
canonical: "https://www.evalgate.com/guides/calibrate-llm-judge"
date-published: "2026-09-02"
last-updated: "2026-09-02"
category: "Evaluation foundations"
keywords: "calibrate LLM judge, LLM judge reliability, judge human agreement"
---

# How to Calibrate an LLM Judge Before Trusting Its Scores

Measure false passes, false failures, stability, and drift against a reviewed human anchor set.

**Question:** How do I calibrate an LLM judge?

## Short answer

Create a versioned set of human-adjudicated examples spanning clear passes, clear failures, and difficult boundaries. Run the judge repeatedly, measure class-specific errors and score stability, revise the rubric rather than the labels, and block automation when calibration drifts.

## Key takeaways

- Calibration needs boundary cases, not just obvious examples.
- False passes and false failures have different product costs.
- Every judge, rubric, and mapping version needs its own evidence.

## Signs this is the problem

- A judge threshold was chosen from intuition or one demo set.
- The same unchanged output receives materially different grades.
- A judge model or rubric changed without rerunning human anchors.

## Step-by-step approach

### 1. Build the anchor set

Use adjudicated examples across intents, score ranges, languages, and known judge failure modes.

### 2. Run repeated trials

Measure parsing failures, score variance, false passes, and false failures under fixed settings.

### 3. Tune the rubric

Separate criteria, add boundary anchors, and make requested evidence explicit without teaching answers from the test set.

### 4. Version and monitor

Freeze the judge configuration and rerun calibration after model, rubric, mapping, or traffic changes.

## What to measure

| Metric | What it measures | How to use it |
| --- | --- | --- |
| True-positive and true-negative rates | Judge performance on human-labeled passes and failures. | Inspect both because aggregate agreement can hide unsafe false passes. |
| Score stability | Variation across repeated grading of identical evidence. | Widen review bands or change the grader when instability crosses the decision margin. |
| Calibration drift | Change in agreement on the frozen anchor set over time. | Pause blocking use when drift is unexplained. |

## Common mistakes

- Calibrating on examples used to write the rubric.
- Reporting correlation without class-specific error rates.
- Changing the score mapping without versioning prior decisions.

## Practical checklist

- [ ] Build the anchor set: Use adjudicated examples across intents, score ranges, languages, and known judge failure modes.
- [ ] Run repeated trials: Measure parsing failures, score variance, false passes, and false failures under fixed settings.
- [ ] Tune the rubric: Separate criteria, add boundary anchors, and make requested evidence explicit without teaching answers from the test set.
- [ ] Version and monitor: Freeze the judge configuration and rerun calibration after model, rubric, mapping, or traffic changes.

## Where EvalGate fits

EvalGate's calibration control plane versions anchor sets and mappings, exposes disagreement, and can prevent incompatible or drifting judge evidence from driving releases.

[Open the calibration manual](https://www.evalgate.com/docs/platform/calibration-control-plane) or [start with the EvalGate quickstart](https://www.evalgate.com/docs/quickstart).

## Frequently asked questions

### What should I do first?

Use adjudicated examples across intents, score ranges, languages, and known judge failure modes

### How should the result be measured?

Judge performance on human-labeled passes and failures. Inspect both because aggregate agreement can hide unsafe false passes.

### When is the change ready to ship?

Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.

## Related guides

- [https://www.evalgate.com/guides/human-review-vs-llm-judge](https://www.evalgate.com/guides/human-review-vs-llm-judge)
- [https://www.evalgate.com/guides/code-based-evals-vs-llm-judge](https://www.evalgate.com/guides/code-based-evals-vs-llm-judge)
- [https://www.evalgate.com/guides/fix-flaky-ai-evals](https://www.evalgate.com/guides/fix-flaky-ai-evals)