Code-Based Evals vs. LLM Judges: Which Should You Use?
Choose deterministic checks, semantic judges, or a layered combination based on what can be verified.
Short answer
Use code-based evals for facts, schemas, tool calls, permissions, state, and other objective contracts. Use an LLM judge for semantic qualities that require interpretation. Strong systems layer them: deterministic failures take precedence, while calibrated judges cover meaning that code cannot express well.
Key takeaways
- Choose the simplest grader that can verify the requirement.
- Never ask a judge to guess state that code can read.
- Calibrate semantic judges against reviewed human labels.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- A model judge grades JSON validity or whether a tool was called.
- String matching rejects correct paraphrases.
- Judge scores change materially when order or style changes.
Step-by-step approach
- 1
Separate objective facts
List requirements that can be checked from structured output, traces, or destination state.
- 2
Write semantic rubrics
For remaining qualities, define distinct criteria with positive and negative anchors.
- 3
Layer the decision
Run deterministic gates first, then semantic grading, with explicit precedence for critical failures.
- 4
Validate the graders
Use labeled examples, disagreement review, and repeated trials to measure each grader's error modes.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Deterministic coverage | Share of objective requirements enforced by code. | Increase this before expanding expensive judging. |
| Judge-human agreement | Agreement on a reviewed anchor set. | Do not make a judge blocking until error rates are acceptable for the risk. |
| Grader stability | Score variation on unchanged inputs. | Investigate unstable thresholds or prompts before blaming the application. |
Common mistakes
- Using lexical similarity as a universal correctness test.
- Letting an LLM judge override a hard safety contract.
- Treating one judge model as ground truth.
Practical checklist
- Separate objective facts: List requirements that can be checked from structured output, traces, or destination state.
- Write semantic rubrics: For remaining qualities, define distinct criteria with positive and negative anchors.
- Layer the decision: Run deterministic gates first, then semantic grading, with explicit precedence for critical failures.
- Validate the graders: Use labeled examples, disagreement review, and repeated trials to measure each grader's error modes.
Frequently asked questions
What should I do first?
List requirements that can be checked from structured output, traces, or destination state
How should the result be measured?
Share of objective requirements enforced by code. Increase this before expanding expensive judging.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.