> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Test agent skills

> Measure skill trigger precision and paired task uplift locally with a bounded runner or recorded observations.

Test whether an agent loads your skill on the right prompts, then compare task
results with and without that skill. The TypeScript and Python source CLIs run
this locally; no EvalGate credential or hosted request is required. Use a build
that includes the `skill` command; source availability does not establish npm or
PyPI release availability.

```bash theme={null}
evalgate skill test --skill ./skills/refund-policy --cases skill-cases.jsonl \
  --runner 'node ./skill-runner.mjs' --trials 3 --format json \
  --min-trigger-f1 0.9 --min-uplift 0.05
```

The command runs your runner once for each case, trial, and variant. It passes the
prompt on stdin and sets `EVALGATE_SKILL_VARIANT` to `with` or `without`, plus
`EVALGATE_SKILL_DIR` to the absolute directory. The command string is split into
an executable and arguments and executed without a shell. Pipes, substitutions,
redirection, and shell expansion do not execute. Put complex setup in a runner
script.

## Define cases and task checks

Each JSONL line has a unique case id, prompt, and expected trigger label. Include
at least ten distinct paired cases; repeating one prompt ten times does not
establish ten independent cases.

```jsonl theme={null}
{"id":"refund-eligible","prompt":"Can I return an unused item purchased yesterday?","expectTrigger":true,"task":{"check":"includes_all","expected":["receipt","30 days"]}}
{"id":"unrelated-weather","prompt":"What is the weather?","expectTrigger":false,"task":{"check":"exit_code","expected":0}}
{"id":"structured-answer","prompt":"Return the refund decision as JSON.","expectTrigger":true,"task":{"check":"json_equals","expected":{"decision":"review"}}}
```

`exit_code` compares the reported task exit code with `expected`. `includes_all`
requires a successful task exit and every expected substring. `json_equals`
requires a successful task exit and structurally equal JSON, ignoring object key
order. JSON numbers must be finite; integer values must be between
`-9007199254740991` and `9007199254740991` so TypeScript and Python compare the
same values. Unsafe integers in expected JSON are refused; unsafe integers in
runner output fail the task check. Encode high-precision identifiers as strings.
Task checks are optional for trigger-only testing; uplift needs task checks.

## Runner contract

Write exactly one JSON line to stdout:

```json theme={null}
{"variant":"with","invokedSkills":["refund-policy"],"output":"Bring your receipt within 30 days.","exitCode":0}
```

`variant` must echo the requested variant. `invokedSkills` lists skills actually
loaded by the agent. The tested skill's name comes from its `SKILL.md` frontmatter
`name`, falling back to the directory name. Mentioning the skill in an answer
does not count as an invocation. The runner process itself must exit zero to
deliver usable evidence; `exitCode` records the task's outcome.

The runner controls isolation: it must remove the skill from the agent's view in
the `without` variant, including global/project skills, agent configuration,
caches, and prior conversation state. EvalGate rejects a `without` observation
that reports loading the tested skill, but cannot prove your runner hid every
copy. Use fresh work directories and fresh sessions for both variants.

Execution defaults to a 30-second timeout and a combined 1 MB stdout/stderr cap.
Set `--timeout-ms` (maximum 600000) and `--max-bytes` (maximum 10000000) explicitly
when needed. Runner errors, malformed JSON, variant mix-ups, and resource-limit
failures cannot produce a passing result. Error reports omit raw output, stderr,
and prompts. Runner code executes with your user permissions and inherited
environment; run only code you trust.

On POSIX systems, EvalGate terminates the runner's process group after every
outcome, including success, to remove remaining child processes. Windows
termination covers the direct runner process; descendant cleanup is not
verified there. A Windows runner must stop its own child processes before
exiting. These limits do not isolate filesystem or network access.

## Example: Claude Code headless runner

This example uses fresh temporary project and configuration directories. Check
the stream contract for your installed Claude Code version before using it for
release evidence. It reads `Skill` tool-use records from stream JSON; it never
infers skill loading from the final answer.

```js theme={null}
// skill-runner.mjs
import { spawn } from "node:child_process";
import { cp, mkdtemp, mkdir, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { basename, join } from "node:path";

const variant = process.env.EVALGATE_SKILL_VARIANT;
if (!["with", "without"].includes(variant)) throw new Error("Invalid variant");
const source = process.env.EVALGATE_SKILL_DIR;
const project = await mkdtemp(join(tmpdir(), "skill-project-"));
const config = await mkdtemp(join(tmpdir(), "skill-config-"));
try {
  if (variant === "with") {
    await mkdir(join(project, ".claude", "skills"), { recursive: true });
    await cp(source, join(project, ".claude", "skills", basename(source)), { recursive: true });
  }
  const prompt = await new Promise(resolve => {
    let text = "";
    process.stdin.on("data", chunk => text += chunk);
    process.stdin.on("end", () => resolve(text));
  });
  const child = spawn("claude", ["-p", "--verbose", "--output-format", "stream-json"], {
    cwd: project, env: { ...process.env, CLAUDE_CONFIG_DIR: config },
    stdio: ["pipe", "pipe", "inherit"], shell: false,
  });
  let stream = "";
  child.stdout.on("data", chunk => stream += chunk);
  child.stdin.end(prompt);
  const code = await new Promise((resolve, reject) => {
    child.on("error", reject);
    child.on("close", resolve);
  });
  const events = stream.trim().split("\n").filter(Boolean).map(line => JSON.parse(line));
  const invokedSkills = events.flatMap(event => event.message?.content ?? [])
    .filter(block => block.type === "tool_use" && block.name === "Skill")
    .map(block => block.input?.skill).filter(name => typeof name === "string");
  const result = events.findLast(event => event.type === "result");
  if (!result) throw new Error("No Claude result record");
  console.log(JSON.stringify({ variant, invokedSkills, output: result.result ?? "",
    exitCode: code === 0 && !result.is_error ? 0 : 1 }));
} finally {
  await rm(project, { recursive: true, force: true });
  await rm(config, { recursive: true, force: true });
}
```

Authenticate Claude Code using your approved mechanism for the isolated config.
Do not copy account secrets into a committed runner. The example can call a paid
model; EvalGate itself performs no model call. Set up a representative fixture
project inside the temporary directory when a task depends on project files.

## Use recorded observations

For harnesses that cannot be scripted, record the same runner fields plus
`caseId` and a one-based `trial`:

```jsonl theme={null}
{"caseId":"refund-eligible","trial":1,"variant":"with","invokedSkills":["refund-policy"],"output":"Bring your receipt within 30 days.","exitCode":0}
{"caseId":"refund-eligible","trial":1,"variant":"without","invokedSkills":[],"output":"Ask support.","exitCode":0}
```

```bash theme={null}
evalgate skill test --skill ./skills/refund-policy --cases skill-cases.jsonl \
  --observations observations.jsonl --trials 3 --format json
```

Supply exactly one observation for every `(caseId, trial, variant)`. Duplicate,
missing, extra, and unknown-case observations are refused rather than scored on
a selected subset. Choose exactly one of `--runner` and `--observations`.

Case and observation inputs must be regular files (no symlinks, devices, or
named pipes), with at most 50 MB per file and 4 MiB per JSONL record. Inputs are
read incrementally with LF or CRLF delimiters; Unicode line and paragraph
separators inside JSON strings remain data. Blank lines count toward the byte
limit without creating retained records. Parsing stops when the case/trial
population bound is exceeded or a record fails validation.

## Interpret and gate results

The report contains trigger precision, recall, F1, false/missed-trigger case ids,
task pass rates for both variants, a paired difference, its seeded 95% percentile
bootstrap interval, and a two-sided exact sign-test p-value. Trials are paired by
case and trial; task results are averaged within each case before bootstrapping
case differences, so repeated trials do not inflate the independent sample size.
The sign test uses nonzero per-case differences and excludes ties.

Undefined trigger metrics are `null`. No task checks means `task: null`, not
zero uplift. The report includes both the `SKILL.md` SHA-256 and a SHA-256 of all
relative filenames and bytes in the skill directory, including scripts and
references. Symlinks are refused. A directory changing during execution invalidates
the evidence. Keep observation files and generated reports outside that directory.

| Exit | Meaning | Action |
| - | - | - |
| 0 | Available evidence clears configured gates | Review the report |
| 1 | Trigger F1 or the uplift interval lower bound is below its threshold | Inspect false/missed triggers and task cases |
| 5 | Invalid options, cases, observations, or files | Correct the local inputs |
| 6 | Fewer than ten independent paired cases | Add distinct labelled cases |
| 7 | Trigger or requested uplift evidence unavailable, or runner failed | Repair the evidence collection |

`--min-uplift 0.05` requires the interval's lower bound to be at least 0.05.
An ungated report describes the observations; it does not prove the skill will
improve every deployment or that a nondeterministic agent always behaves alike.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.