LLM judge orchestration in EvalGate

How EvalGate’s judge system works: registry-backed model selection, multi-judge composition, credibility metrics, and enterprise policy enforcement.

An LLM judge is an evaluation backed by a language model that produces a structured result — a score, a pass/fail verdict, and reasoning tied to specific signals — rather than a raw string response. EvalGate’s judge system is not a simple model switcher. It is a control plane: registry-backed model selection, saved presets, deterministic aggregation across multiple judges, per-judge evidence, disagreement handling, and enterprise policy enforcement.

BYOK requirement

LLM judges use bring your own provider key (BYOK). EvalGate does not include model inference credits: your organization connects a provider or gateway credential, and that provider bills the judge calls directly. The EvalGate API key authenticates your request to EvalGate; it does not authorize the model call. Before testing a judge, follow Model providers and BYOK to connect the credential, verify provider health and model availability, and set provider-side and EvalGate budget controls.

Current judge guidance

As of June 24, 2026, treat LLM judges as versioned measurement instruments, not neutral truth machines:

Prefer deterministic rules for exact checks before asking a model to judge language.
Use score-model judges only with explicit rubrics, bounded numeric ranges, pass thresholds, and fixed sampling settings.
Calibrate each judge prompt against human-labeled examples before trusting it in a release gate. Track true positive rate, true negative rate, parse failures, and disagreement.
Include few-shot examples when the rubric is nuanced, and keep those examples versioned with the judge.
Use pairwise judging when “which answer is better” is more reliable than assigning an absolute score.
Escalate high-disagreement, high-cost, safety-critical, or policy-sensitive cases to human review.
Store the exact prompt, model, parser, output schema, raw result, cost, latency, and threshold used for every score.

OpenAI’s current grader taxonomy includes string checks, text similarity, score-model graders, and Python code execution. LangSmith’s evaluator model separates human, code, LLM-as-judge, and pairwise evaluation techniques. Phoenix recommends validating custom LLM judges against benchmark datasets before relying on them for subjective quality. Sources: OpenAI graders, LangSmith evaluation concepts, and Phoenix custom LLM evaluator benchmark workflow.

Calibration runbook

Do this before letting a judge block merges:

Create a labeled calibration set

Start with at least 30 human-labeled examples per critical task or failure mode. Include passing cases, obvious failures, borderline cases, and examples where earlier judges disagreed.

Lock the judge artifact

Version the rubric, prompt, model, parser, threshold, few-shot examples, sampling settings, and expected JSON schema. A judge config change is a measurement change, not a routine code refactor.

Measure reliability

Run the judge against the labeled set and inspect true positive rate, true negative rate, parse failure rate, confidence interval, disagreement rate, latency, and cost.

Set gate policy

Use deterministic assertions for hard contracts, then gate judge-backed checks only when reliability clears your configured threshold. Route high-disagreement or low-confidence cases to human review.

Recalibrate on drift

Re-run calibration after changing judge model, rubric, threshold, dataset slice, or application behavior. EvalGate marks incompatible judge configurations so trend lines do not imply false continuity.

evalgate.config.json

{
  "judge": {
    "bootstrapSeed": 42,
    "tprMin": 0.70,
    "tnrMin": 0.70,
    "minLabeledSamples": 30
  },
  "review": {
    "escalateOnDisagreement": true,
    "highRiskFailureModes": ["safety", "privacy", "tool_misuse"]
  }
}

Do not use a fresh LLM judge as the only release gate. Use it as an evidence source until calibration proves it can separate passing and failing cases for your task.

The registry and presets

Instead of hardcoding provider names and model versions throughout your evaluation code, EvalGate maintains a registry of available judges and a library of presets that group judge configurations for common use cases. The registry tracks each judge’s reliability tier (trusted, stable, or experimental), availability, and whether it is allowed under your organization’s policy. Presets let you pick a named configuration — default, fast, quality, or economy — and let EvalGate resolve the right models without scattering model names across your codebase.

# List available judges in the registry
npx @evalgate/sdk judge registry

# List saved presets
npx @evalgate/sdk judge presets

TypeScript

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const registry = await client.llmJudge.listRegistry();
const presets = await client.llmJudge.listPresets();

Python

from evalgate_sdk import AIEvalClient

client = AIEvalClient(api_key="sk-...")

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

Start with a preset. Presets are configured with stable, trusted judges and locked temperature settings for deterministic scoring. Use the registry to explore alternatives once you have a baseline.

Testing a judge configuration

Before using a judge in production evaluations, test its behavior on a representative input with testConfig. This runs the judge against a real input/output pair and returns the full result including score, reasoning, signals, and metadata.

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const result = await client.llmJudge.testConfig({
  provider: 'openai',
  model: 'gpt-5.2-chat-latest',
  promptTemplate:
    'Return strict JSON with score, passed, reasoning, and signals.',
  judges: [
    {
      id: 'primary',
      type: 'llm',
      provider: 'openai',
      model: 'gpt-5.2-chat-latest',
    },
    {
      id: 'fallback',
      type: 'llm',
      provider: 'anthropic',
      model: 'claude-sonnet-4-20250514',
    },
  ],
  aggregation: 'weighted',
  input: 'Cancel my subscription',
  output: "I've canceled your plan effective today.",
  behavior: 'tool_use',
  taskType: 'support',
});

console.log(result.result.score, result.result.reasoning);

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import TestLLMJudgeConfigParams

client = AIEvalClient(api_key="sk-...")

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.2-chat-latest",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.2-chat-latest",
            },
            {
                "id": "fallback",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-sonnet-4-20250514",
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.result.score, result.result.reasoning)

You can also test a judge from the CLI with fine-grained control over provider, model, aggregation strategy, and prompt template:

npx @evalgate/sdk judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Score the output for correctness and completeness using structured JSON." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."

Multi-judge composition

A single judge is sufficient for most evaluations. Add more judges when you need higher reliability, fallback coverage, or explicit disagreement analysis. When you compose multiple judges, you choose an aggregation strategy that determines how individual judge verdicts combine into a final result:

Strategy	Behavior
`all_pass`	The case passes only when every judge passes it
`any_pass`	The case passes when at least one judge passes it
`weighted`	Scores are combined using per-judge weights
`primary_fallback`	Use the primary judge result; fall back to the next if the primary fails to produce a valid result
`escalate_on_disagreement`	Flag the case for human review when judges disagree instead of resolving the conflict automatically

Prefer weighted or escalate_on_disagreement over averaging. Averaging hides disagreement — escalate_on_disagreement surfaces it as a signal that the case may need human review or a clearer rubric.

A practical progression

Start with one trusted judge

Choose a preset from the judge workspace or CLI. Keep temperature locked for deterministic judging. Inspect case-level results before adding more judges.

Add a rule judge for exact checks

For schema-sensitive or safety-critical assertions, add a rule-based judge alongside your LLM judge. Rule judges are fast, cheap, and deterministic.

Add a second LLM judge only when needed

Add a second LLM judge when you have evidence of instability or when disagreement analysis is valuable — not preemptively. Use escalate_on_disagreement to make disagreements visible.

What a judge result exposes

Each judge result in EvalGate carries more than a score. The full result includes:

Field	Description
`score`	Normalized score from 0 to 1
`passed`	Boolean pass/fail verdict
`reasoning`	Natural language explanation tied to structured signals
`signals`	Specific behaviors or properties the judge detected
`provider`	Model provider and version provenance
`latency`	Time taken for the judge call in milliseconds
`tokens`	Token usage for the judge call
`retries`	Number of retries needed to get a valid result
`parseStatus`	Whether the response parsed successfully or required fallback handling
`disagreement`	Whether this judge’s verdict differs from other judges in a multi-judge run

This provenance lets you reproduce any score, debug unexpected verdicts, and audit judge behavior over time.

Judge credibility

A judge that produces unreliable scores is worse than no judge — it gives you false confidence. EvalGate tracks credibility metrics for every configured judge:

True positive rate (TPR) and true negative rate (TNR)

TPR measures how often the judge correctly identifies failures. TNR measures how often it correctly identifies passing cases. Both are gated in evalgate.config.json:

{
  "judge": {
    "bootstrapSeed": 42,
    "tprMin": 0.70,
    "tnrMin": 0.70,
    "minLabeledSamples": 30
  }
}

When discriminative power (TPR + TNR − 1) falls to 0.05 or below, EvalGate skips score correction and exits the gate with code 8 (WARN) instead of silently using a biased score.

Bootstrap confidence intervals

EvalGate computes 95% bootstrap confidence intervals on pass rates. With fewer than 30 labeled samples, CI computation is skipped and a caution note appears in the judgeCredibility block of the JSON report. With fewer than 5 samples, results are suppressed entirely.Set bootstrapSeed in your config for deterministic CI runs.

Reliability tiers

Each judge in the registry is assigned a reliability tier based on its operational track record:

Tier	Criteria
`trusted`	Parse failure rate < 1%, disagreement rate < 10%
`stable`	Parse failure rate < 5%, disagreement rate < 25%
`experimental`	All others — requires explicit org allowlist approval

Disagreement analysis

In multi-judge runs, EvalGate measures score spread (standard deviation, range, min/max), pass/fail splits across judges, and outlier judges deviating more than 0.3 from the group mean. Cases with high disagreement (range ≥ 0.4 or any pass/fail split) are flagged for human review — they are often the most informative cases in your dataset.

Enterprise controls

Before EvalGate sends any data to an external judge provider, it enforces your organization’s policy rules:

Provider allowlists — Only providers explicitly approved for your org can receive evaluation data
Cost caps — Judge calls that exceed configured cost thresholds are blocked or rerouted
Latency budgets — Slow judges can be automatically replaced by faster fallbacks
PII redaction — Personally identifiable information is scrubbed from inputs before any external call
Audit logging — Every judge execution is logged with full provenance for compliance review

A judge that receives un-redacted PII produces results that are not just incorrect but potentially harmful from a compliance standpoint. Enable PII redaction in your org settings before configuring external judge providers.

Where judges appear in the product

Run composer — Choose preset, configure judges, set aggregation strategy, review estimated cost and latency
Run summary — See judge set used, disagreement rate across the run, and baseline score deltas
Case review — Inspect per-judge reasoning and signal breakdown for individual test cases
Comparison view — Compare disagreement, cost, latency, and instability across two runs
Registry — Browse reliability tier, availability, and policy status for every available judge

Get Started

Core Concepts

Guides

SDK Reference

Platform

Llm judge

LLM judge orchestration in EvalGate

BYOK requirement

Current judge guidance

Calibration runbook

The registry and presets

Testing a judge configuration

Multi-judge composition

A practical progression

What a judge result exposes

Judge credibility

Enterprise controls

Where judges appear in the product

​LLM judge orchestration in EvalGate

​BYOK requirement

​Current judge guidance

​Calibration runbook

​The registry and presets

​Testing a judge configuration

​Multi-judge composition

​A practical progression

​What a judge result exposes

​Judge credibility

​Enterprise controls

​Where judges appear in the product

LLM judge orchestration in EvalGate

BYOK requirement

Current judge guidance

Calibration runbook

The registry and presets

Testing a judge configuration

Multi-judge composition

A practical progression

What a judge result exposes

Judge credibility

Enterprise controls

Where judges appear in the product