How to Set Pass/Fail Thresholds for AI Evaluations
Turn quality scores into stable release decisions with risk tiers, uncertainty bands, and protected slices.
Short answer
Set thresholds from reviewed baseline performance, measurement error, and the cost of false passes versus false failures. Use non-compensating floors for critical behaviors, a review band for uncertainty, and version thresholds before seeing candidate results.
Key takeaways
- Thresholds encode risk tolerance, not mathematical truth.
- Critical failures need separate floors or zero-tolerance rules.
- Use a review band when scorer uncertainty overlaps the boundary.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- A round number became the threshold without baseline evidence.
- Teams lower the bar after seeing a candidate fail.
- One high score compensates for a forbidden outcome.
Step-by-step approach
- 1
Classify the risk
Separate advisory, important, and release-blocking behaviors before choosing numbers.
- 2
Measure the baseline
Estimate score distributions, repeated-run variance, and judge errors on reviewed cases.
- 3
Define decision bands
Set pass, manual-review, and fail regions plus hard rules for critical outcomes.
- 4
Version and revisit
Record the rationale and change thresholds only with new evidence, not to rescue a release.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Decision margin | Distance between observed performance and the relevant threshold. | Treat narrow margins as uncertain when measurement variance is similar. |
| False-pass cost | Impact and frequency of bad behavior incorrectly accepted. | Use it to tighten high-risk thresholds or require deterministic proof. |
| False-fail cost | Safe candidates incorrectly blocked by the evaluation. | Use review bands and better graders instead of silently lowering the floor. |
Common mistakes
- Choosing thresholds after candidate results are visible.
- Using the same threshold for every risk class.
- Ignoring scorer confidence and sample size.
Practical checklist
- Classify the risk: Separate advisory, important, and release-blocking behaviors before choosing numbers.
- Measure the baseline: Estimate score distributions, repeated-run variance, and judge errors on reviewed cases.
- Define decision bands: Set pass, manual-review, and fail regions plus hard rules for critical outcomes.
- Version and revisit: Record the rationale and change thresholds only with new evidence, not to rescue a release.
Frequently asked questions
What should I do first?
Separate advisory, important, and release-blocking behaviors before choosing numbers
How should the result be measured?
Distance between observed performance and the relevant threshold. Treat narrow margins as uncertain when measurement variance is similar.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.