How to Fix Flaky and Nondeterministic AI Evaluations
Separate application variance, grader variance, infrastructure failure, and threshold sensitivity before adding retries.
Short answer
Replay identical evidence and vary one component at a time to locate the variance: application output, model judge, external dependency, parser, or threshold. Use repeated samples and uncertainty-aware decisions; retries should expose instability, not convert failures into passes.
Key takeaways
- Measure variance before choosing a retry count.
- Pin every input and version that can affect the result.
- Use review bands when uncertainty overlaps the threshold.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- The same stored output receives different judge labels.
- Rerunning until green is accepted release practice.
- Infrastructure errors are mixed with quality failures.
Step-by-step approach
- 1
Freeze the evidence
Pin cases, outputs, prompts, models, parameters, tools, source data, graders, mappings, and thresholds.
- 2
Isolate each layer
Replay stored outputs through graders, then repeat application generation separately to locate variance.
- 3
Quantify uncertainty
Measure disagreement, score spread, parse failures, and decision flips near the boundary.
- 4
Redesign the decision
Prefer deterministic checks, clearer rubrics, repeated estimates, review bands, or quarantined unstable cases.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Decision flip rate | Identical conditions producing different pass, warn, or fail decisions. | Make blocking gates stable enough for their risk. |
| Layer-specific variance | Output and score variation attributed to each component. | Fix the dominant layer rather than increasing all retries. |
| Invalid-run rate | Infrastructure, parsing, or missing-evidence runs that cannot support a quality decision. | Report these as invalid, never as passes. |
Common mistakes
- Rerunning only failed candidates until they pass.
- Averaging invalid and valid results together.
- Widening thresholds without revisiting the product risk.
Practical checklist
- Freeze the evidence: Pin cases, outputs, prompts, models, parameters, tools, source data, graders, mappings, and thresholds.
- Isolate each layer: Replay stored outputs through graders, then repeat application generation separately to locate variance.
- Quantify uncertainty: Measure disagreement, score spread, parse failures, and decision flips near the boundary.
- Redesign the decision: Prefer deterministic checks, clearer rubrics, repeated estimates, review bands, or quarantined unstable cases.
Frequently asked questions
What should I do first?
Pin cases, outputs, prompts, models, parameters, tools, source data, graders, mappings, and thresholds
How should the result be measured?
Identical conditions producing different pass, warn, or fail decisions. Make blocking gates stable enough for their risk.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.