How to Calibrate an LLM Judge Before Trusting Its Scores
Measure false passes, false failures, stability, and drift against a reviewed human anchor set.
Short answer
Create a versioned set of human-adjudicated examples spanning clear passes, clear failures, and difficult boundaries. Run the judge repeatedly, measure class-specific errors and score stability, revise the rubric rather than the labels, and block automation when calibration drifts.
Key takeaways
- Calibration needs boundary cases, not just obvious examples.
- False passes and false failures have different product costs.
- Every judge, rubric, and mapping version needs its own evidence.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- A judge threshold was chosen from intuition or one demo set.
- The same unchanged output receives materially different grades.
- A judge model or rubric changed without rerunning human anchors.
Step-by-step approach
- 1
Build the anchor set
Use adjudicated examples across intents, score ranges, languages, and known judge failure modes.
- 2
Run repeated trials
Measure parsing failures, score variance, false passes, and false failures under fixed settings.
- 3
Tune the rubric
Separate criteria, add boundary anchors, and make requested evidence explicit without teaching answers from the test set.
- 4
Version and monitor
Freeze the judge configuration and rerun calibration after model, rubric, mapping, or traffic changes.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| True-positive and true-negative rates | Judge performance on human-labeled passes and failures. | Inspect both because aggregate agreement can hide unsafe false passes. |
| Score stability | Variation across repeated grading of identical evidence. | Widen review bands or change the grader when instability crosses the decision margin. |
| Calibration drift | Change in agreement on the frozen anchor set over time. | Pause blocking use when drift is unexplained. |
Common mistakes
- Calibrating on examples used to write the rubric.
- Reporting correlation without class-specific error rates.
- Changing the score mapping without versioning prior decisions.
Practical checklist
- Build the anchor set: Use adjudicated examples across intents, score ranges, languages, and known judge failure modes.
- Run repeated trials: Measure parsing failures, score variance, false passes, and false failures under fixed settings.
- Tune the rubric: Separate criteria, add boundary anchors, and make requested evidence explicit without teaching answers from the test set.
- Version and monitor: Freeze the judge configuration and rerun calibration after model, rubric, mapping, or traffic changes.
Frequently asked questions
What should I do first?
Use adjudicated examples across intents, score ranges, languages, and known judge failure modes
How should the result be measured?
Judge performance on human-labeled passes and failures. Inspect both because aggregate agreement can hide unsafe false passes.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.