Test agent skills

Test whether an agent loads your skill on the right prompts, then compare task results with and without that skill. The TypeScript and Python source CLIs run this locally; no EvalGate credential or hosted request is required. Use a build that includes the skill command; source availability does not establish npm or PyPI release availability.

evalgate skill test --skill ./skills/refund-policy --cases skill-cases.jsonl \
  --runner 'node ./skill-runner.mjs' --trials 3 --format json \
  --min-trigger-f1 0.9 --min-uplift 0.05

The command runs your runner once for each case, trial, and variant. It passes the prompt on stdin and sets EVALGATE_SKILL_VARIANT to with or without, plus EVALGATE_SKILL_DIR to the absolute directory. The command string is split into an executable and arguments and executed without a shell. Pipes, substitutions, redirection, and shell expansion do not execute. Put complex setup in a runner script.

Define cases and task checks

Each JSONL line has a unique case id, prompt, and expected trigger label. Include at least ten distinct paired cases; repeating one prompt ten times does not establish ten independent cases.

{"id":"refund-eligible","prompt":"Can I return an unused item purchased yesterday?","expectTrigger":true,"task":{"check":"includes_all","expected":["receipt","30 days"]}}
{"id":"unrelated-weather","prompt":"What is the weather?","expectTrigger":false,"task":{"check":"exit_code","expected":0}}
{"id":"structured-answer","prompt":"Return the refund decision as JSON.","expectTrigger":true,"task":{"check":"json_equals","expected":{"decision":"review"}}}

exit_code compares the reported task exit code with expected. includes_all requires a successful task exit and every expected substring. json_equals requires a successful task exit and structurally equal JSON, ignoring object key order. JSON numbers must be finite; integer values must be between -9007199254740991 and 9007199254740991 so TypeScript and Python compare the same values. Unsafe integers in expected JSON are refused; unsafe integers in runner output fail the task check. Encode high-precision identifiers as strings. Task checks are optional for trigger-only testing; uplift needs task checks.

Runner contract

Write exactly one JSON line to stdout:

{"variant":"with","invokedSkills":["refund-policy"],"output":"Bring your receipt within 30 days.","exitCode":0}

variant must echo the requested variant. invokedSkills lists skills actually loaded by the agent. The tested skill’s name comes from its SKILL.md frontmatter name, falling back to the directory name. Mentioning the skill in an answer does not count as an invocation. The runner process itself must exit zero to deliver usable evidence; exitCode records the task’s outcome. The runner controls isolation: it must remove the skill from the agent’s view in the without variant, including global/project skills, agent configuration, caches, and prior conversation state. EvalGate rejects a without observation that reports loading the tested skill, but cannot prove your runner hid every copy. Use fresh work directories and fresh sessions for both variants. Execution defaults to a 30-second timeout and a combined 1 MB stdout/stderr cap. Set --timeout-ms (maximum 600000) and --max-bytes (maximum 10000000) explicitly when needed. Runner errors, malformed JSON, variant mix-ups, and resource-limit failures cannot produce a passing result. Error reports omit raw output, stderr, and prompts. Runner code executes with your user permissions and inherited environment; run only code you trust. On POSIX systems, EvalGate terminates the runner’s process group after every outcome, including success, to remove remaining child processes. Windows termination covers the direct runner process; descendant cleanup is not verified there. A Windows runner must stop its own child processes before exiting. These limits do not isolate filesystem or network access.

Example: Claude Code headless runner

This example uses fresh temporary project and configuration directories. Check the stream contract for your installed Claude Code version before using it for release evidence. It reads Skill tool-use records from stream JSON; it never infers skill loading from the final answer.

// skill-runner.mjs
import { spawn } from "node:child_process";
import { cp, mkdtemp, mkdir, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { basename, join } from "node:path";

const variant = process.env.EVALGATE_SKILL_VARIANT;
if (!["with", "without"].includes(variant)) throw new Error("Invalid variant");
const source = process.env.EVALGATE_SKILL_DIR;
const project = await mkdtemp(join(tmpdir(), "skill-project-"));
const config = await mkdtemp(join(tmpdir(), "skill-config-"));
try {
  if (variant === "with") {
    await mkdir(join(project, ".claude", "skills"), { recursive: true });
    await cp(source, join(project, ".claude", "skills", basename(source)), { recursive: true });
  }
  const prompt = await new Promise(resolve => {
    let text = "";
    process.stdin.on("data", chunk => text += chunk);
    process.stdin.on("end", () => resolve(text));
  });
  const child = spawn("claude", ["-p", "--verbose", "--output-format", "stream-json"], {
    cwd: project, env: { ...process.env, CLAUDE_CONFIG_DIR: config },
    stdio: ["pipe", "pipe", "inherit"], shell: false,
  });
  let stream = "";
  child.stdout.on("data", chunk => stream += chunk);
  child.stdin.end(prompt);
  const code = await new Promise((resolve, reject) => {
    child.on("error", reject);
    child.on("close", resolve);
  });
  const events = stream.trim().split("\n").filter(Boolean).map(line => JSON.parse(line));
  const invokedSkills = events.flatMap(event => event.message?.content ?? [])
    .filter(block => block.type === "tool_use" && block.name === "Skill")
    .map(block => block.input?.skill).filter(name => typeof name === "string");
  const result = events.findLast(event => event.type === "result");
  if (!result) throw new Error("No Claude result record");
  console.log(JSON.stringify({ variant, invokedSkills, output: result.result ?? "",
    exitCode: code === 0 && !result.is_error ? 0 : 1 }));
} finally {
  await rm(project, { recursive: true, force: true });
  await rm(config, { recursive: true, force: true });
}

Authenticate Claude Code using your approved mechanism for the isolated config. Do not copy account secrets into a committed runner. The example can call a paid model; EvalGate itself performs no model call. Set up a representative fixture project inside the temporary directory when a task depends on project files.

Use recorded observations

For harnesses that cannot be scripted, record the same runner fields plus caseId and a one-based trial:

{"caseId":"refund-eligible","trial":1,"variant":"with","invokedSkills":["refund-policy"],"output":"Bring your receipt within 30 days.","exitCode":0}
{"caseId":"refund-eligible","trial":1,"variant":"without","invokedSkills":[],"output":"Ask support.","exitCode":0}

evalgate skill test --skill ./skills/refund-policy --cases skill-cases.jsonl \
  --observations observations.jsonl --trials 3 --format json

Supply exactly one observation for every (caseId, trial, variant). Duplicate, missing, extra, and unknown-case observations are refused rather than scored on a selected subset. Choose exactly one of --runner and --observations. Case and observation inputs must be regular files (no symlinks, devices, or named pipes), with at most 50 MB per file and 4 MiB per JSONL record. Inputs are read incrementally with LF or CRLF delimiters; Unicode line and paragraph separators inside JSON strings remain data. Blank lines count toward the byte limit without creating retained records. Parsing stops when the case/trial population bound is exceeded or a record fails validation.

Interpret and gate results

The report contains trigger precision, recall, F1, false/missed-trigger case ids, task pass rates for both variants, a paired difference, its seeded 95% percentile bootstrap interval, and a two-sided exact sign-test p-value. Trials are paired by case and trial; task results are averaged within each case before bootstrapping case differences, so repeated trials do not inflate the independent sample size. The sign test uses nonzero per-case differences and excludes ties. Undefined trigger metrics are null. No task checks means task: null, not zero uplift. The report includes both the SKILL.md SHA-256 and a SHA-256 of all relative filenames and bytes in the skill directory, including scripts and references. Symlinks are refused. A directory changing during execution invalidates the evidence. Keep observation files and generated reports outside that directory.

Exit	Meaning	Action
0	Available evidence clears configured gates	Review the report
1	Trigger F1 or the uplift interval lower bound is below its threshold	Inspect false/missed triggers and task cases
5	Invalid options, cases, observations, or files	Correct the local inputs
6	Fewer than ten independent paired cases	Add distinct labelled cases
7	Trigger or requested uplift evidence unavailable, or runner failed	Repair the evidence collection

--min-uplift 0.05 requires the interval’s lower bound to be at least 0.05. An ungated report describes the observations; it does not prove the skill will improve every deployment or that a nondeterministic agent always behaves alike.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Test agent skills

Define cases and task checks

Runner contract

Example: Claude Code headless runner

Use recorded observations

Interpret and gate results

​Define cases and task checks

​Runner contract

​Example: Claude Code headless runner

​Use recorded observations

​Interpret and gate results

Define cases and task checks

Runner contract

Example: Claude Code headless runner

Use recorded observations

Interpret and gate results