> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Llm judge

# LLM judge orchestration in EvalGate

> How EvalGate's judge system works: registry-backed model selection, multi-judge composition, credibility metrics, and enterprise policy enforcement.

An LLM judge is an evaluation backed by a language model that produces a structured result — a score, a pass/fail verdict, and reasoning tied to specific signals — rather than a raw string response. EvalGate's judge system is not a simple model switcher. It is a control plane: registry-backed model selection, saved presets, deterministic aggregation across multiple judges, per-judge evidence, disagreement handling, and enterprise policy enforcement.

## BYOK requirement

LLM judges use **bring your own provider key (BYOK)**. EvalGate does not include
model inference credits: your organization connects a provider or gateway
credential, and that provider bills the judge calls directly. The EvalGate API
key authenticates your request to EvalGate; it does not authorize the model
call.

Before testing a judge, follow [Model providers and BYOK](/docs/platform/model-providers-byok)
to connect the credential, verify provider health and model availability, and
set provider-side and EvalGate budget controls.

## Current judge guidance

As of June 24, 2026, treat LLM judges as versioned measurement instruments, not neutral truth machines:

* Prefer deterministic rules for exact checks before asking a model to judge language.
* Use score-model judges only with explicit rubrics, bounded numeric ranges, pass thresholds, and fixed sampling settings.
* Calibrate each judge prompt against human-labeled examples before trusting it in a release gate. Track true positive rate, true negative rate, parse failures, and disagreement.
* Include few-shot examples when the rubric is nuanced, and keep those examples versioned with the judge.
* Use pairwise judging when "which answer is better" is more reliable than assigning an absolute score.
* Escalate high-disagreement, high-cost, safety-critical, or policy-sensitive cases to human review.
* Store the exact prompt, model, parser, output schema, raw result, cost, latency, and threshold used for every score.

OpenAI's current grader taxonomy includes string checks, text similarity, score-model graders, and Python code execution. LangSmith's evaluator model separates human, code, LLM-as-judge, and pairwise evaluation techniques. Phoenix recommends validating custom LLM judges against benchmark datasets before relying on them for subjective quality.

Sources: [OpenAI graders](https://platform.openai.com/docs/guides/graders), [LangSmith evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts), and [Phoenix custom LLM evaluator benchmark workflow](https://arize.com/docs/phoenix/cookbook/evaluation/creating-a-custom-llm-evaluator-with-a-benchmark-dataset).

## Calibration runbook

Do this before letting a judge block merges:

<Steps>
  <Step title="Create a labeled calibration set">
    Start with at least 30 human-labeled examples per critical task or failure mode. Include passing cases, obvious failures, borderline cases, and examples where earlier judges disagreed.
  </Step>

  <Step title="Lock the judge artifact">
    Version the rubric, prompt, model, parser, threshold, few-shot examples, sampling settings, and expected JSON schema. A judge config change is a measurement change, not a routine code refactor.
  </Step>

  <Step title="Measure reliability">
    Run the judge against the labeled set and inspect true positive rate, true negative rate, parse failure rate, confidence interval, disagreement rate, latency, and cost.
  </Step>

  <Step title="Set gate policy">
    Use deterministic assertions for hard contracts, then gate judge-backed checks only when reliability clears your configured threshold. Route high-disagreement or low-confidence cases to human review.
  </Step>

  <Step title="Recalibrate on drift">
    Re-run calibration after changing judge model, rubric, threshold, dataset slice, or application behavior. EvalGate marks incompatible judge configurations so trend lines do not imply false continuity.
  </Step>
</Steps>

```json evalgate.config.json theme={null} theme={null}
{
  "judge": {
    "bootstrapSeed": 42,
    "tprMin": 0.70,
    "tnrMin": 0.70,
    "minLabeledSamples": 30
  },
  "review": {
    "escalateOnDisagreement": true,
    "highRiskFailureModes": ["safety", "privacy", "tool_misuse"]
  }
}
```

<Warning>
  Do not use a fresh LLM judge as the only release gate. Use it as an evidence source until calibration proves it can separate passing and failing cases for your task.
</Warning>

## The registry and presets

Instead of hardcoding provider names and model versions throughout your evaluation code, EvalGate maintains a **registry** of available judges and a library of **presets** that group judge configurations for common use cases.

The registry tracks each judge's reliability tier (`trusted`, `stable`, or `experimental`), availability, and whether it is allowed under your organization's policy. Presets let you pick a named configuration — `default`, `fast`, `quality`, or `economy` — and let EvalGate resolve the right models without scattering model names across your codebase.

```bash theme={null} theme={null}
# List available judges in the registry
npx @evalgate/sdk judge registry

# List saved presets
npx @evalgate/sdk judge presets
```

```ts TypeScript theme={null} theme={null}
import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const registry = await client.llmJudge.listRegistry();
const presets = await client.llmJudge.listPresets();
```

```python Python theme={null} theme={null}
from evalgate_sdk import AIEvalClient

client = AIEvalClient(api_key="sk-...")

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()
```

<Tip>
  Start with a preset. Presets are configured with stable, trusted judges and locked temperature settings for deterministic scoring. Use the registry to explore alternatives once you have a baseline.
</Tip>

## Testing a judge configuration

Before using a judge in production evaluations, test its behavior on a representative input with `testConfig`. This runs the judge against a real input/output pair and returns the full result including score, reasoning, signals, and metadata.

<CodeGroup>
  ```ts TypeScript theme={null} theme={null}
  import { AIEvalClient } from '@evalgate/sdk';

  const client = AIEvalClient.init();

  const result = await client.llmJudge.testConfig({
    provider: 'openai',
    model: 'gpt-5.2-chat-latest',
    promptTemplate:
      'Return strict JSON with score, passed, reasoning, and signals.',
    judges: [
      {
        id: 'primary',
        type: 'llm',
        provider: 'openai',
        model: 'gpt-5.2-chat-latest',
      },
      {
        id: 'fallback',
        type: 'llm',
        provider: 'anthropic',
        model: 'claude-sonnet-4-20250514',
      },
    ],
    aggregation: 'weighted',
    input: 'Cancel my subscription',
    output: "I've canceled your plan effective today.",
    behavior: 'tool_use',
    taskType: 'support',
  });

  console.log(result.result.score, result.result.reasoning);
  ```

  ```python Python theme={null} theme={null}
  from evalgate_sdk import AIEvalClient
  from evalgate_sdk.types import TestLLMJudgeConfigParams

  client = AIEvalClient(api_key="sk-...")

  result = await client.llm_judge.test_config(
      TestLLMJudgeConfigParams(
          provider="openai",
          model="gpt-5.2-chat-latest",
          prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
          judges=[
              {
                  "id": "primary",
                  "type": "llm",
                  "provider": "openai",
                  "model": "gpt-5.2-chat-latest",
              },
              {
                  "id": "fallback",
                  "type": "llm",
                  "provider": "anthropic",
                  "model": "claude-sonnet-4-20250514",
              },
          ],
          aggregation="weighted",
          input="Cancel my subscription",
          output="I've canceled your plan effective today.",
          behavior="tool_use",
          task_type="support",
      )
  )

  print(result.result.score, result.result.reasoning)
  ```
</CodeGroup>

You can also test a judge from the CLI with fine-grained control over provider, model, aggregation strategy, and prompt template:

```bash theme={null} theme={null}
npx @evalgate/sdk judge test \
  --provider openai \
  --model gpt-5.2-chat-latest \
  --judge openai:gpt-5.2-chat-latest \
  --judge anthropic:claude-sonnet-4-20250514 \
  --aggregation weighted \
  --prompt-template "Score the output for correctness and completeness using structured JSON." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."
```

## Multi-judge composition

A single judge is sufficient for most evaluations. Add more judges when you need higher reliability, fallback coverage, or explicit disagreement analysis.

When you compose multiple judges, you choose an **aggregation strategy** that determines how individual judge verdicts combine into a final result:

| Strategy                   | Behavior                                                                                            |
| -------------------------- | --------------------------------------------------------------------------------------------------- |
| `all_pass`                 | The case passes only when every judge passes it                                                     |
| `any_pass`                 | The case passes when at least one judge passes it                                                   |
| `weighted`                 | Scores are combined using per-judge weights                                                         |
| `primary_fallback`         | Use the primary judge result; fall back to the next if the primary fails to produce a valid result  |
| `escalate_on_disagreement` | Flag the case for human review when judges disagree instead of resolving the conflict automatically |

<Note>
  Prefer `weighted` or `escalate_on_disagreement` over averaging. Averaging hides disagreement — `escalate_on_disagreement` surfaces it as a signal that the case may need human review or a clearer rubric.
</Note>

### A practical progression

<Steps>
  <Step title="Start with one trusted judge">
    Choose a preset from the judge workspace or CLI. Keep temperature locked for deterministic judging. Inspect case-level results before adding more judges.
  </Step>

  <Step title="Add a rule judge for exact checks">
    For schema-sensitive or safety-critical assertions, add a rule-based judge alongside your LLM judge. Rule judges are fast, cheap, and deterministic.
  </Step>

  <Step title="Add a second LLM judge only when needed">
    Add a second LLM judge when you have evidence of instability or when disagreement analysis is valuable — not preemptively. Use `escalate_on_disagreement` to make disagreements visible.
  </Step>
</Steps>

## What a judge result exposes

Each judge result in EvalGate carries more than a score. The full result includes:

| Field          | Description                                                                 |
| -------------- | --------------------------------------------------------------------------- |
| `score`        | Normalized score from 0 to 1                                                |
| `passed`       | Boolean pass/fail verdict                                                   |
| `reasoning`    | Natural language explanation tied to structured signals                     |
| `signals`      | Specific behaviors or properties the judge detected                         |
| `provider`     | Model provider and version provenance                                       |
| `latency`      | Time taken for the judge call in milliseconds                               |
| `tokens`       | Token usage for the judge call                                              |
| `retries`      | Number of retries needed to get a valid result                              |
| `parseStatus`  | Whether the response parsed successfully or required fallback handling      |
| `disagreement` | Whether this judge's verdict differs from other judges in a multi-judge run |

This provenance lets you reproduce any score, debug unexpected verdicts, and audit judge behavior over time.

## Judge credibility

A judge that produces unreliable scores is worse than no judge — it gives you false confidence. EvalGate tracks credibility metrics for every configured judge:

<AccordionGroup>
  <Accordion title="True positive rate (TPR) and true negative rate (TNR)">
    TPR measures how often the judge correctly identifies failures. TNR measures how often it correctly identifies passing cases. Both are gated in `evalgate.config.json`:

    ```json theme={null} theme={null}
    {
      "judge": {
        "bootstrapSeed": 42,
        "tprMin": 0.70,
        "tnrMin": 0.70,
        "minLabeledSamples": 30
      }
    }
    ```

    When discriminative power (TPR + TNR − 1) falls to 0.05 or below, EvalGate skips score correction and exits the gate with code `8 (WARN)` instead of silently using a biased score.
  </Accordion>

  <Accordion title="Bootstrap confidence intervals">
    EvalGate computes 95% bootstrap confidence intervals on pass rates. With fewer than 30 labeled samples, CI computation is skipped and a caution note appears in the `judgeCredibility` block of the JSON report. With fewer than 5 samples, results are suppressed entirely.

    Set `bootstrapSeed` in your config for deterministic CI runs.
  </Accordion>

  <Accordion title="Reliability tiers">
    Each judge in the registry is assigned a reliability tier based on its operational track record:

    | Tier           | Criteria                                              |
    | -------------- | ----------------------------------------------------- |
    | `trusted`      | Parse failure rate \< 1%, disagreement rate \< 10%    |
    | `stable`       | Parse failure rate \< 5%, disagreement rate \< 25%    |
    | `experimental` | All others — requires explicit org allowlist approval |
  </Accordion>

  <Accordion title="Disagreement analysis">
    In multi-judge runs, EvalGate measures score spread (standard deviation, range, min/max), pass/fail splits across judges, and outlier judges deviating more than 0.3 from the group mean. Cases with high disagreement (`range ≥ 0.4` or any pass/fail split) are flagged for human review — they are often the most informative cases in your dataset.
  </Accordion>
</AccordionGroup>

## Enterprise controls

Before EvalGate sends any data to an external judge provider, it enforces your organization's policy rules:

* **Provider allowlists** — Only providers explicitly approved for your org can receive evaluation data
* **Cost caps** — Judge calls that exceed configured cost thresholds are blocked or rerouted
* **Latency budgets** — Slow judges can be automatically replaced by faster fallbacks
* **PII redaction** — Personally identifiable information is scrubbed from inputs before any external call
* **Audit logging** — Every judge execution is logged with full provenance for compliance review

<Warning>
  A judge that receives un-redacted PII produces results that are not just incorrect but potentially harmful from a compliance standpoint. Enable PII redaction in your org settings before configuring external judge providers.
</Warning>

## Where judges appear in the product

* **Run composer** — Choose preset, configure judges, set aggregation strategy, review estimated cost and latency
* **Run summary** — See judge set used, disagreement rate across the run, and baseline score deltas
* **Case review** — Inspect per-judge reasoning and signal breakdown for individual test cases
* **Comparison view** — Compare disagreement, cost, latency, and instability across two runs
* **Registry** — Browse reliability tier, availability, and policy status for every available judge