> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Write evals with your coding agent
> Let EvalGate understand the product first, confirm the working theory, then turn real preferred and undesired responses into quarantined eval drafts.
# Understand the product before writing evals
You should not have to explain tracing, install an OpenTelemetry collector, or
fill out a long questionnaire to get a useful first eval. EvalGate starts by
reading the product surface and teaching its working theory back to you. You
correct that theory once, then use real examples to define what good behavior
means.
This understand-and-author workflow is experimental. Install the Skill
directly from the repository. The TypeScript and Python CLIs are available
locally from the current source tree. REST, mounted web authoring,
marketplace publication, controlled Codex/Claude/Copilot compatibility
proof, and package and release-environment evidence are still pending.
## Install the authoring skill
Install the portable skill in a compatible coding agent:
```bash theme={null}
npx skills add https://github.com/evalgate/ai-evaluation-platform --skill evalgate-authoring
```
Then ask:
> Add evals to this product. First understand what it does from repository
> evidence, show me your working theory and one important question, and do not
> apply or promote anything without asking.
The skill calls the same EvalGate CLI and schemas used by local development and
CI. It does not maintain a separate eval format.
## Get a useful working theory first
The agent begins with a read-only preview:
```bash theme={null}
npx @evalgate/sdk understand --format json
```
It should respond as soon as there is enough evidence for a useful first read.
It should not make you wait for a comprehensive scan. The response should look
roughly like this:
> **Working theory:** This is a support assistant for customers asking about
> orders and refunds. The visible output is a policy-grounded answer with a
> concrete next step.
>
> **Observed:** The refund route, support prompt, and fixture data all reference
> a 30-day standard window.
>
> **Hypotheses:** The suite should probably protect policy accuracy, directness,
> and escalation when an exception is possible.
>
> **Confidence:** Medium. The core flow is clear, but no approved response style
> or exception behavior is saved yet.
>
> **One question:** When completeness and speed conflict, which should the
> customer notice first?
The labels matter:
* **Observed** means repository content, tests, fixtures, saved behavior, or an
example you supplied supports the statement.
* **Hypothesis** means the agent inferred product intent, a failure mode, or a
preferred response that you have not confirmed.
A plausible failure is not a real failure. An agent-written answer is not
trusted product truth just because it sounds good.
The preview performs no file writes, network calls, model calls, Git changes,
baseline changes, or promotions.
## Confirm one product context
Answer the single question that would most change the evals, then review the
proposed context. After explicit approval, save it:
```bash theme={null}
npx @evalgate/sdk understand --apply
```
EvalGate writes the reviewed context to `evalgate.product.json`. That file is
the canonical statement of product intent, evidence, observations, hypotheses,
confidence, and open questions.
Claude, Cursor, Codex, Copilot, and other native agent instruction files should
only point to it:
```text theme={null}
Read evalgate.product.json before authoring evals.
```
Do not copy the product description or quality rules into every agent file.
One canonical artifact prevents stale and contradictory instructions.
## Describe quality with real contrasts
After confirming or correcting the product context, give the agent:
* a realistic user input;
* the preferred response;
* a response to avoid;
* what is observably wrong with the avoided response;
* any product-specific priority or behavior that must never occur.
Use specific problems such as "invented refund eligibility" or "ignored the
requested output format." Avoid broad labels such as "bad quality."
If the agent proposes a preferred response, failure mode, or behavior to
protect, it must remain labeled as a hypothesis until you edit or approve it.
The agent proposes a versioned `evalgate.quality.json`:
```json theme={null}
{
"schemaVersion": 1,
"name": "Support assistant",
"domainContext": "Answers customers asking about refunds.",
"priorities": [
"Answer directly",
"Use policy-grounded language"
],
"forbiddenBehaviors": [
"Never invent refund eligibility"
],
"rubricText": "Pass when the answer gives a supported next step.",
"objectiveTargets": [
{
"metric": "policy_accuracy",
"operator": "gte",
"value": 0.95
}
],
"examples": [
{
"id": "refund-window",
"input": "Can I get a refund after 45 days?",
"preferredOutput": "Our standard window is 30 days. I can help check whether an exception applies.",
"undesiredOutput": "Yes, your refund is guaranteed.",
"problem": "Invented refund eligibility",
"dimensionValues": { "window": "outside" }
}
],
"dimensions": {
"window": ["inside", "outside"]
}
}
```
The profile is editable repository data. It does not contain a hidden prompt or
provider-specific judge configuration.
Top-level `dimensions` are a coverage vocabulary. An example's
`dimensionValues` records only the scenario values that its input and outputs
actually establish. Profile synthesis creates one draft per contrastive example;
it does not manufacture a Cartesian set by relabeling an unchanged example. Add
another real contrastive example when you want coverage for another value.
## Preview before writing
```bash theme={null}
npx @evalgate/sdk synthesize \
--profile evalgate.quality.json \
--format json
```
Preview reports:
* the normalized profile hash;
* how many contrastive examples were supplied;
* which quarantined cases would be created;
* whether the output is new or would be replaced;
* the exact next command.
Profile preview performs no file writes, network calls, model calls, Git
mutations, baseline changes, or promotions.
## Create quarantined drafts
After reviewing the profile and preview:
```bash theme={null}
npx @evalgate/sdk synthesize \
--profile evalgate.quality.json \
--apply
```
The command writes `.evalgate/golden/synthetic.jsonl`. Every row has
`lifecycleState: "synthetic"` and records which profile and example produced
it. Applying drafts does not make them a merge blocker.
Review generated cases separately before running local promotion. Baseline
acceptance and Git changes are also separate actions.
## Add real usage when it helps
Real usage is an optional next source of examples, not a setup requirement.
EvalGate uses these technical terms:
| Product-language idea | Technical term |
| -------------------------------------------- | ----------------------- |
| One complete user interaction | Trace |
| A model, retrieval, or tool step inside it | Span |
| A standard service that forwards run records | OpenTelemetry collector |
Connect automatic capture when you need production discovery, latency and cost
details, or multi-step debugging. Until then, pasted interactions, saved run
files, API responses, and hand-written contrastive examples can all begin the
evaluation loop.
When you have representative real interactions:
1. Label each pass or fail and note the first observable problem.
2. Let product-specific failure categories emerge from the examples.
3. Fix the underlying product behavior when possible.
4. Create one binary evaluator per recurring subjective problem.
5. Validate model-based judges against independent human labels before gating.
Synthetic cases fill coverage gaps. They do not replace human review or prove
that production behavior is covered.