Understand the product before writing evals

You should not have to explain tracing, install an OpenTelemetry collector, or fill out a long questionnaire to get a useful first eval. EvalGate starts by reading the product surface and teaching its working theory back to you. You correct that theory once, then use real examples to define what good behavior means.

This understand-and-author workflow is experimental. Install the Skill directly from the repository. The TypeScript and Python CLIs are available locally from the current source tree. REST, mounted web authoring, marketplace publication, controlled Codex/Claude/Copilot compatibility proof, and package and release-environment evidence are still pending.

Install the authoring skill

Install the portable skill in a compatible coding agent:

npx skills add https://github.com/evalgate/ai-evaluation-platform --skill evalgate-authoring

Then ask:

Add evals to this product. First understand what it does from repository evidence, show me your working theory and one important question, and do not apply or promote anything without asking.

The skill calls the same EvalGate CLI and schemas used by local development and CI. It does not maintain a separate eval format.

Get a useful working theory first

The agent begins with a read-only preview:

npx @evalgate/sdk understand --format json

It should respond as soon as there is enough evidence for a useful first read. It should not make you wait for a comprehensive scan. The response should look roughly like this:

Working theory: This is a support assistant for customers asking about orders and refunds. The visible output is a policy-grounded answer with a concrete next step. Observed: The refund route, support prompt, and fixture data all reference a 30-day standard window. Hypotheses: The suite should probably protect policy accuracy, directness, and escalation when an exception is possible. Confidence: Medium. The core flow is clear, but no approved response style or exception behavior is saved yet. One question: When completeness and speed conflict, which should the customer notice first?

The labels matter:

Observed means repository content, tests, fixtures, saved behavior, or an example you supplied supports the statement.
Hypothesis means the agent inferred product intent, a failure mode, or a preferred response that you have not confirmed.

A plausible failure is not a real failure. An agent-written answer is not trusted product truth just because it sounds good. The preview performs no file writes, network calls, model calls, Git changes, baseline changes, or promotions.

Confirm one product context

Answer the single question that would most change the evals, then review the proposed context. After explicit approval, save it:

npx @evalgate/sdk understand --apply

EvalGate writes the reviewed context to evalgate.product.json. That file is the canonical statement of product intent, evidence, observations, hypotheses, confidence, and open questions. Claude, Cursor, Codex, Copilot, and other native agent instruction files should only point to it:

Read evalgate.product.json before authoring evals.

Do not copy the product description or quality rules into every agent file. One canonical artifact prevents stale and contradictory instructions.

Describe quality with real contrasts

After confirming or correcting the product context, give the agent:

a realistic user input;
the preferred response;
a response to avoid;
what is observably wrong with the avoided response;
any product-specific priority or behavior that must never occur.

Use specific problems such as “invented refund eligibility” or “ignored the requested output format.” Avoid broad labels such as “bad quality.” If the agent proposes a preferred response, failure mode, or behavior to protect, it must remain labeled as a hypothesis until you edit or approve it. The agent proposes a versioned evalgate.quality.json:

{
  "schemaVersion": 1,
  "name": "Support assistant",
  "domainContext": "Answers customers asking about refunds.",
  "priorities": [
    "Answer directly",
    "Use policy-grounded language"
  ],
  "forbiddenBehaviors": [
    "Never invent refund eligibility"
  ],
  "rubricText": "Pass when the answer gives a supported next step.",
  "objectiveTargets": [
    {
      "metric": "policy_accuracy",
      "operator": "gte",
      "value": 0.95
    }
  ],
  "examples": [
    {
      "id": "refund-window",
      "input": "Can I get a refund after 45 days?",
      "preferredOutput": "Our standard window is 30 days. I can help check whether an exception applies.",
      "undesiredOutput": "Yes, your refund is guaranteed.",
      "problem": "Invented refund eligibility",
      "dimensionValues": { "window": "outside" }
    }
  ],
  "dimensions": {
    "window": ["inside", "outside"]
  }
}

The profile is editable repository data. It does not contain a hidden prompt or provider-specific judge configuration. Top-level dimensions are a coverage vocabulary. An example’s dimensionValues records only the scenario values that its input and outputs actually establish. Profile synthesis creates one draft per contrastive example; it does not manufacture a Cartesian set by relabeling an unchanged example. Add another real contrastive example when you want coverage for another value.

Preview before writing

npx @evalgate/sdk synthesize \
  --profile evalgate.quality.json \
  --format json

Preview reports:

the normalized profile hash;
how many contrastive examples were supplied;
which quarantined cases would be created;
whether the output is new or would be replaced;
the exact next command.

Profile preview performs no file writes, network calls, model calls, Git mutations, baseline changes, or promotions.

Create quarantined drafts

After reviewing the profile and preview:

npx @evalgate/sdk synthesize \
  --profile evalgate.quality.json \
  --apply

The command writes .evalgate/golden/synthetic.jsonl. Every row has lifecycleState: "synthetic" and records which profile and example produced it. Applying drafts does not make them a merge blocker. Review generated cases separately before running local promotion. Baseline acceptance and Git changes are also separate actions.

Add real usage when it helps

Real usage is an optional next source of examples, not a setup requirement. EvalGate uses these technical terms:

Product-language idea	Technical term
One complete user interaction	Trace
A model, retrieval, or tool step inside it	Span
A standard service that forwards run records	OpenTelemetry collector

Connect automatic capture when you need production discovery, latency and cost details, or multi-step debugging. Until then, pasted interactions, saved run files, API responses, and hand-written contrastive examples can all begin the evaluation loop. When you have representative real interactions:

Label each pass or fail and note the first observable problem.
Let product-specific failure categories emerge from the examples.
Fix the underlying product behavior when possible.
Create one binary evaluator per recurring subjective problem.
Validate model-based judges against independent human labels before gating.

Synthetic cases fill coverage gaps. They do not replace human review or prove that production behavior is covered.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Write evals with your coding agent

Understand the product before writing evals

Install the authoring skill

Get a useful working theory first

Confirm one product context

Describe quality with real contrasts

Preview before writing

Create quarantined drafts

Add real usage when it helps

​Understand the product before writing evals

​Install the authoring skill

​Get a useful working theory first

​Confirm one product context

​Describe quality with real contrasts

​Preview before writing

​Create quarantined drafts

​Add real usage when it helps

Understand the product before writing evals

Install the authoring skill

Get a useful working theory first

Confirm one product context

Describe quality with real contrasts

Preview before writing

Create quarantined drafts

Add real usage when it helps