> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Llm judge

# LLM Judge API — evaluate and orchestrate judges

> List judge configurations, evaluate LLM outputs, retrieve results, and measure judge alignment with true positive and true negative rates.

The LLM Judge API lets you programmatically run AI-powered quality assessments on any input/output pair. Use it to build automated evaluation pipelines, measure judge credibility against human labels, and retrieve detailed scoring breakdowns — including reasoning, signals, and confidence metrics.

<Warning>
  Judge calls require [bring your own provider key (BYOK)](/docs/platform/model-providers-byok). Your EvalGate bearer token authenticates this API request, while the provider or gateway credential authorizes the model call. EvalGate does not bundle inference credits; the connected provider bills usage directly.
</Warning>

## GET /api/llm-judge/configs — list judge configurations

Returns the judge configurations available in your organization.

```bash theme={null} theme={null}
curl https://evalgate.com/api/llm-judge/configs \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Response

```json theme={null} theme={null}
{
  "configs": [
    {
      "id": 7,
      "name": "Support quality committee",
      "provider": "openai",
      "model": "gpt-4o",
      "aggregation": "weighted",
      "createdAt": "2026-03-01T09:00:00.000Z"
    }
  ]
}
```

***

## POST /api/llm-judge/evaluate — evaluate an output

Submits an input/output pair for evaluation by a judge. You can reference a saved configuration by `configId`, or pass a `judgeConfig` object inline.

```bash theme={null} theme={null}
curl https://evalgate.com/api/llm-judge/evaluate \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "configId": 7,
    "input": "Cancel my subscription",
    "output": "I have canceled your plan effective today. You will retain access through the end of the billing period."
  }'
```

### Request body

<ParamField body="input" type="string" required>
  The original prompt or user query that was sent to your LLM.
</ParamField>

<ParamField body="output" type="string" required>
  The LLM response to evaluate.
</ParamField>

<ParamField body="configId" type="integer">
  ID of a saved judge configuration. Use this or `judgeConfig` — not both.
</ParamField>

<ParamField body="judgeConfig" type="object">
  Inline judge configuration. Use this when you do not have a saved config.

  <Expandable title="judgeConfig fields">
    <ParamField body="provider" type="string">LLM provider: `openai`, `anthropic`, etc.</ParamField>
    <ParamField body="model" type="string">Model name to use as the judge.</ParamField>
    <ParamField body="promptTemplate" type="string">Prompt template instructing the judge how to score.</ParamField>
  </Expandable>
</ParamField>

### Response

```json theme={null} theme={null}
{
  "result": {
    "provider": "openai",
    "model": "gpt-4o",
    "score": 92,
    "passed": true,
    "reasoning": "The response correctly fulfills the cancellation request and clearly communicates the effective date and access period.",
    "signals": ["clear_confirmation", "billing_period_noted"],
    "latency": 1320,
    "tokens": 210,
    "retries": 0,
    "parseStatus": "ok",
    "disagreement": null
  }
}
```

<ResponseField name="result.provider" type="string">LLM provider used for the judge call.</ResponseField>
<ResponseField name="result.model" type="string">Model used for the judge call.</ResponseField>
<ResponseField name="result.score" type="integer">Quality score from 0–100.</ResponseField>
<ResponseField name="result.passed" type="boolean">Whether the output met the passing threshold defined in the judge config.</ResponseField>
<ResponseField name="result.reasoning" type="string">The judge's natural-language explanation of the score.</ResponseField>

<ResponseField name="result.signals" type="array">
  List of signal strings the judge identified — positive indicators, failure patterns, or flagged behaviors.
</ResponseField>

<ResponseField name="result.latency" type="integer">Time in milliseconds the judge call took.</ResponseField>
<ResponseField name="result.tokens" type="integer">Total tokens consumed by the judge call.</ResponseField>
<ResponseField name="result.retries" type="integer">Number of retries the judge performed before returning a parseable result.</ResponseField>

<ResponseField name="result.parseStatus" type="string">
  Whether the judge's response was parsed cleanly. `ok` means structured output was extracted successfully.
</ResponseField>

<ResponseField name="result.disagreement" type="object | null">
  When using a multi-judge committee, this field contains disagreement metrics across judges. `null` for single-judge evaluations.
</ResponseField>

***

## GET /api/llm-judge/results — get evaluation results

Returns stored evaluation results for review, filtering, or export.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/llm-judge/results?configId=7&limit=50" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Query parameters

<ParamField query="configId" type="integer">
  Filter results to a specific judge configuration.
</ParamField>

<ParamField query="limit" type="integer">
  Maximum number of results to return. Defaults to 50.
</ParamField>

<ParamField query="offset" type="integer">
  Pagination offset. Defaults to 0.
</ParamField>

***

## POST /api/llm-judge/alignment

Check judge alignment against human labels and return agreement metrics.

Measures how well a judge agrees with human labels by computing true positive rate (TPR) and true negative rate (TNR) against your annotation dataset. Run this after collecting a sufficient set of human labels via the [Annotations API](/docs/api/annotations).

```bash theme={null} theme={null}
curl https://evalgate.com/api/llm-judge/alignment \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "configId": 7,
    "annotationTaskId": 12
  }'
```

### Request body

<ParamField body="configId" type="integer" required>
  ID of the judge configuration to measure.
</ParamField>

<ParamField body="annotationTaskId" type="integer" required>
  ID of the annotation task containing the human labels to compare against.
</ParamField>

### Response

```json theme={null} theme={null}
{
  "alignment": {
    "configId": 7,
    "annotationTaskId": 12,
    "tpr": 0.91,
    "tnr": 0.87,
    "accuracy": 0.89,
    "sampleSize": 120,
    "computedAt": "2026-03-15T11:00:00.000Z"
  }
}
```

<ResponseField name="alignment.tpr" type="number">True positive rate — fraction of human-labeled passes that the judge also marked as passed.</ResponseField>
<ResponseField name="alignment.tnr" type="number">True negative rate — fraction of human-labeled failures that the judge also marked as failed.</ResponseField>
<ResponseField name="alignment.accuracy" type="number">Overall agreement rate between the judge and human labels.</ResponseField>
<ResponseField name="alignment.sampleSize" type="integer">Number of labeled items used for this alignment calculation.</ResponseField>

<Tip>
  Target a TPR and TNR above 0.85 before using a judge to gate CI runs. Lower alignment means the judge may block good outputs or miss real failures.
</Tip>