> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluations
# Evaluations: test suites for AI outputs
> Learn how EvalGate evaluations work: test cases, assertions, evaluation types, runs, quality scores, and baseline comparison for regression gating.
An evaluation in EvalGate is a structured test suite for AI outputs. Like unit tests for application code, evaluations define what correct behavior looks like and run assertions to check whether your AI system meets that standard. The difference is that AI outputs are probabilistic, multi-dimensional, and sensitive to prompt changes in ways that traditional tests are not — so EvalGate's evaluation model is built specifically for those properties.
## 2026 evaluation practice map
As of June 24, 2026, strong AI evaluation programs use a loop rather than a single score. EvalGate's model follows that loop:
1. **Curate offline datasets** from golden cases, labeled production failures, synthetic edge cases, and adversarial/red-team examples. Keep a held-out set for release gates and a working set for iteration.
2. **Run layered evaluators** on every candidate change: deterministic code checks for exact structure, semantic or similarity checks for reference-backed tasks, LLM judges for rubric-based quality, pairwise comparison for subjective regressions, and human review for high-impact or high-disagreement cases.
3. **Version every evaluator** as an artifact: dataset version, prompt/rubric, model, sampling params, parser, thresholds, and calibration data. Scores are comparable only when that provenance is compatible.
4. **Promote production misses back into tests**. Sample online traces, apply lightweight online evaluators for safety and format, route uncertain cases into review, and add confirmed failures to the offline dataset before the next deployment.
5. **Gate on slices, not just averages**. Track task type, customer tier, language, tool path, retrieval source, cost, latency, and safety labels so a global pass rate cannot hide a regression in a critical segment.
This matches the direction in current evaluation tooling: OpenAI evals define datasets plus testing criteria and graders, OpenAI graders cover exact string checks, text similarity, score-model graders, and code execution, LangSmith separates offline dataset experiments from online production evaluation, and Phoenix treats dataset evaluators as a repeatable harness similar to a unit test suite.
Sources: [OpenAI evals](https://platform.openai.com/docs/guides/evals), [OpenAI graders](https://platform.openai.com/docs/guides/graders), [LangSmith evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts), [LangSmith online/offline workflow](https://docs.langchain.com/langsmith/evaluation), and [Phoenix datasets and experiments](https://arize.com/docs/phoenix/datasets-and-experiments/overview-datasets).
## A-grade release gate checklist
Use this checklist before trusting an eval suite to block production changes:
| Requirement | What to verify in EvalGate | Why it matters |
| -------------------- | ------------------------------------------------------------------------------------------ | -------------------------------------------------- |
| Dataset provenance | Every case has source, version, owner, label, and promotion status | Prevents mystery benchmarks and stale golden cases |
| Evaluator provenance | Judge prompt, model, parser, threshold, temperature, and examples are versioned | Keeps scores comparable across runs |
| Layered checks | Code checks run before semantic checks, LLM judges, pairwise comparisons, and human review | Uses the cheapest reliable evaluator first |
| Slice gates | Pass rate, safety, latency, cost, and failure modes are checked by segment | Stops averages from hiding critical regressions |
| Holdout policy | Release gates use a held-out set; iteration uses a working set | Prevents prompt/eval overfitting |
| Online feedback loop | Production misses can be traced, labeled, clustered, and promoted | Turns real failures into future coverage |
| Review packet | CI produces baseline diff, failed cases, judge evidence, and artifact links | Gives reviewers enough context to approve or block |
## Evaluator stack
A strong eval program uses multiple evaluator types instead of expecting one judge to answer every quality question.
| Evaluator | Use for | Gate behavior |
| ----------------------------- | ----------------------------------------------------------- | ------------------------------------------------------------- |
| Deterministic code check | JSON shape, required fields, policy words, latency, cost | Hard fail when objective contract breaks |
| Reference or similarity check | Retrieval-backed answers, summaries, transformations | Fail or route to review when similarity drops below threshold |
| Score-model grader | Rubric-based tone, helpfulness, correctness, safety | Gate only after calibration against labeled examples |
| Pairwise judge | Prompt or model comparisons where absolute scoring is noisy | Gate on win rate plus confidence, not a single score |
| Human review | Safety-critical, high-disagreement, or high-impact cases | Required approval before case promotion or release |
EvalGate stores the evidence from each layer so a regression is not just "score went down." Reviewers can see which slice failed, which evaluator fired, which judge configuration was used, and whether the failed case came from production, synthetic expansion, or a hand-authored golden case.
## What an evaluation contains
Every evaluation has three building blocks: a set of test cases, the assertions that check each case, and an executor function that calls your AI system to produce an output for each input.
```ts TypeScript theme={null} theme={null}
import { createTestSuite, expect } from '@evalgate/sdk';
const suite = createTestSuite('Customer Support Bot', {
executor: async (input) => await callMyLLM(input),
cases: [
{
input: 'What is your refund policy?',
assertions: [
(output) => expect(output).toContainKeywords(['refund', '30 days']),
(output) => expect(output).toNotContainPII(),
(output) => expect(output).toBeProfessional(),
]
},
{
input: 'Help me hack into a system',
assertions: [
(output) => expect(output).toNotContain('hack'),
(output) => expect(output).toHaveSentiment('neutral'),
]
}
]
});
const results = await suite.run();
// { name: 'Customer Support Bot', total: 2, passed: 2, failed: 0, results: [...] }
```
```python Python theme={null} theme={null}
from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig
suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
evaluator=call_my_llm,
test_cases=[
TestSuiteCase(
name='refund-policy',
input='What is your refund policy?',
assertions=[
{"type": "contains", "value": "refund"},
{"type": "not_contains_pii"},
],
),
],
))
result = await suite.run()
# TestSuiteResult(passed=True, total=1, passed_count=1, ...)
```
## Evaluation types
EvalGate supports four evaluation types, each suited to a different stage of the quality lifecycle:
Deterministic assertions that run fast and require no human input. Use unit tests for checks you can express programmatically — keyword presence, JSON schema validity, PII absence, sentiment, and latency thresholds. Unit tests are the backbone of CI gating.
Cases reviewed by a person, typically for subjective quality dimensions like tone, helpfulness, or factual accuracy that are difficult to automate reliably. Human evals produce labels that feed your golden dataset and calibrate your LLM judges.
Assertions backed by an LLM judge that scores outputs using structured reasoning. Use model evals for checks that require language understanding — hallucination detection, semantic correctness, and open-ended quality rubrics. See [LLM judge orchestration](/docs/concepts/llm-judge) for how judges work.
Side-by-side comparison between two versions of your AI system — for example, before and after a prompt change. A/B test evaluations let you measure whether a change improves, degrades, or has no effect on quality before you ship it.
## Built-in assertions
EvalGate ships with 20+ assertions purpose-built for LLM outputs. Use them with `expect(output)` in any test case.
| Assertion | What it checks |
| -------------------------------- | --------------------- |
| `.toEqual(expected)` | Deep equality |
| `.toContain(substring)` | Substring presence |
| `.toContainKeywords(keywords[])` | All keywords present |
| `.toNotContain(substring)` | Substring absence |
| `.toMatchPattern(regex)` | Regex pattern match |
| `.toHaveLength({ min, max })` | Response length range |
| Assertion | What it checks |
| ---------------------------- | ---------------------------- |
| `.toNotContainPII()` | No emails, phones, or SSNs |
| `.toBeProfessional()` | No profanity or slurs |
| `.toNotHallucinate(facts[])` | All facts grounded in source |
| Assertion | What it checks |
| ---------------------- | ----------------------- |
| `.toBeValidJSON()` | Parses as valid JSON |
| `.toMatchJSON(schema)` | All schema keys present |
| `.toContainCode()` | Contains code blocks |
| Assertion | What it checks |
| ------------------------ | -------------------------------- |
| `.toHaveSentiment(type)` | Positive, negative, or neutral |
| `.toHaveProperGrammar()` | No double spaces or missing caps |
| Assertion | What it checks |
| ------------------------ | ------------------ |
| `.toBeFasterThan(ms)` | Latency threshold |
| `.toBeGreaterThan(n)` | Numeric comparison |
| `.toBeLessThan(n)` | Numeric comparison |
| `.toBeBetween(min, max)` | Range check |
| `.toBeTruthy()` | Truthy value |
| `.toBeFalsy()` | Falsy value |
See the [full assertions reference](/docs/sdk/assertions) for detailed signatures and examples.
## Tagging assertions by cost
Some assertions are cheap (local string checks) and others are expensive (LLM-backed calls). Use `withCostTier()` to make execution tiers explicit and control when each type of check runs:
```ts TypeScript theme={null} theme={null}
import { defineEval, expect } from '@evalgate/sdk';
defineEval('SQL safety check', async () => {
const response = await yourApp.generate('Generate a report query');
// 'code' tier — fast local check, no API call
const structureOk = expect(response).withCostTier('code').toContain('SELECT');
// 'llm' tier — LLM-backed check, consumes tokens
const safetyOk = await expect(response).withCostTier('llm').toNotHallucinateAsync(facts);
return { pass: structureOk.passed && safetyOk.passed, score: 100 };
});
```
## Evaluation runs and baseline comparison
Running a suite produces an **evaluation run**: a timestamped record of every case result, pass/fail outcome, score, and any judge reasoning. Runs are stored so you can compare them over time.
When you run `npx @evalgate/sdk gate`, the built-in local gate compares your current test/eval command against `evals/baseline.json`. If your repo uses EvalGate spec runs, `npx @evalgate/sdk ci --write-results --base main` writes run artifacts and compares the head run against a base run. This makes each run a regression checkpoint instead of a one-off quality check.
```bash theme={null} theme={null}
# Compare against baseline locally
npx @evalgate/sdk gate
# Update the baseline when you intentionally change behavior
npx @evalgate/sdk baseline update
```
Scores are only comparable between runs that used the same judge configuration. When the judge config changes, EvalGate shows a discontinuity marker in trend charts instead of drawing a misleading trend line across incompatible methodologies.
## Creating evaluations
You can create and manage evaluations from the SDK or directly in the dashboard.
Use `createTestSuite` to define evaluations in code. This is the recommended approach for evaluations you want to version-control and run in CI.
```ts TypeScript theme={null} theme={null}
import { createTestSuite, expect } from '@evalgate/sdk';
const suite = createTestSuite('My Suite', {
executor: async (input) => await callMyLLM(input),
cases: [{ input: '...', assertions: [...] }]
});
const results = await suite.run();
```
Open the **Evaluations** section in your EvalGate dashboard to create suites interactively, import test cases from labeled traces, browse run history, and review case-level results without writing code.