> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cli

# EvalGate CLI command reference

> Complete reference for all EvalGate CLI commands: setup, gates, CI integration, trace labeling, failure analysis, judge orchestration, and auto loops.

The EvalGate CLI is the fastest way to run regression gates, analyze failure patterns, and automate prompt improvement without leaving your terminal. Run TypeScript CLI commands with `npx @evalgate/sdk <command>` for zero-install usage, or add `@evalgate/sdk` to your project and run the same commands through your package manager.

<Tabs>
  <Tab title="TypeScript (npx)">
    ```bash theme={null} theme={null}
    npm install @evalgate/sdk
    npx @evalgate/sdk <command>
    ```
  </Tab>

  <Tab title="Python">
    ```bash theme={null} theme={null}
    pip install evalgate-sdk
    evalgate <command>
    ```
  </Tab>
</Tabs>

## Navigate by outcome

Both CLIs expose the same versioned capability map. Start here when you know the
job you need to accomplish but not the internal command name:

```bash theme={null} theme={null}
# Human-readable map
evalgate capabilities

# One workflow with web routes, native commands, operation IDs, artifacts, and next step
evalgate capabilities playground

# Stable machine discovery for coding agents
evalgate capabilities --format json
```

The complete map of governed core workflows covers setup, evaluation, Playground, Prompt Hub,
Dataset Hub, first-party evaluation packs, repository intelligence, experiments, scorer development, logs, continuous evaluation,
failure topics, dashboards, Copilot, provider onboarding, red-team testing,
remote runners, deployable assets, governance, and release. TypeScript and
Python emit identical JSON from the generated contract. Every mapped workflow
names its web route, public operation IDs, native automation command, artifacts,
and next governed step; generation fails when any required workflow disappears.

## Stable exit codes

| Code  | Meaning                               |
| ----- | ------------------------------------- |
| `0`   | Success                               |
| `1`   | Score below threshold                 |
| `2`   | Regression                            |
| `3`   | Policy violation                      |
| `4`   | API or network error                  |
| `5`   | Invalid arguments                     |
| `6`   | Insufficient sample size              |
| `7`   | Weak evidence                         |
| `8`   | Warning-level regression              |
| `9`   | Failure-mode threshold exceeded       |
| `10`  | Judge credibility untrusted           |
| `11`  | Local IO error                        |
| `12`  | Baseline checksum mismatch            |
| `99`  | Internal error                        |
| `130` | Cancelled by the user or orchestrator |

## Call any public API operation

The TypeScript CLI's `api` command consumes the generated public operation registry. This gives automation a single command for every operation in the API reference without falling back to private routes or database helpers. Treat it as an escape hatch for advanced automation; `evalgate capabilities` and the native workflow commands are the primary navigation surface.

```bash theme={null} theme={null}
# Inspect stable operation IDs and their methods, paths, and parameter contracts
npx @evalgate/sdk api --list-operations --json

# Read one page
npx @evalgate/sdk api get_evaluations \
  --query limit=25 \
  --query offset=0 \
  --json

# Make an idempotent mutation
npx @evalgate/sdk api post_evaluations \
  --body '{"name":"Support quality"}' \
  --idempotency-key create-support-quality-v1 \
  --json

# Traverse only pagination declared by the operation contract
npx @evalgate/sdk api get_evaluations \
  --paginate \
  --limit 50 \
  --max-pages 20 \
  --json
```

Use `--path name=value`, `--query name=value`, and `--header name=value` more than once when an operation declares multiple inputs. `--body` accepts inline JSON or `@path/to/body.json`. Machine mode writes the exact successful API envelope to stdout; failures write a stable JSON error containing `code`, `message`, `status`, `requestId`, and `exitCode`. `Ctrl-C` aborts the active request and exits `130`.

## Setup and initialization

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk init — plan and scaffold a project">
    Detects Node, Python, polyglot, and nested workspace projects and defaults to a non-mutating plan. Review the files, commands, network boundary, and rollback instructions, then apply explicitly. Init creates a pending baseline without executing tests or accepting results, and it requires no account.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk init
    npx @evalgate/sdk init --format json
    npx @evalgate/sdk init --apply
    npx @evalgate/sdk baseline update
    ```

    Supported first-class handlers are npm, pnpm, the legacy `yarn` alias,
    Yarn Classic, Yarn Modern, Bun, Deno, pip, uv, Poetry, Pipenv, PDM,
    Conda, Mamba, Hatch, Pixi, and explicit custom JSON argv. Conflicting
    JavaScript or Python/environment markers stop the plan. Repeat
    `--package-handler <id>` for an explicit polyglot composition; gate and
    baseline update execute every runnable handler from its owning root.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk init --package-handler bun --package-handler uv
    ```

    `baseline update` accepts only successful current evidence.
    Then commit the generated files and push to trigger your first CI gate:

    ```bash theme={null} theme={null}
    git add evals/ .github/workflows/evalgate-gate.yml evalgate.config.json
    git push
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk doctor — environment diagnostics">
    Diagnoses local project readiness without cloud access in quick mode. Omit
    `--quick` when you also want account and platform connectivity checks.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk doctor --quick
    ```
  </Accordion>
</AccordionGroup>

## Gate and CI

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk gate — run the regression gate locally">
    Compares your current test results against the stored baseline and exits `1` if any metric regresses.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk gate
    ```

    Output format options:

    ```bash theme={null} theme={null}
    npx @evalgate/sdk gate --format github   # CI step summary and job annotations
    npx @evalgate/sdk gate --format json     # Machine-readable JSON output
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk ci — one-command CI gate">
    Discovers eval specs, runs them, writes results, and compares against a base run when `--base` is provided. Add `--impacted-only` to run only specs affected by the current diff.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk ci --format github --write-results --base main
    ```

    Full GitHub Actions workflow:

    ```yaml theme={null} theme={null}
    name: EvalGate CI
    on: [push, pull_request]
    jobs:
      evalgate:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v4
          - uses: actions/setup-node@v4
          - run: npm ci
          - run: npx @evalgate/sdk ci --format github --write-results --base main
          - uses: actions/upload-artifact@v4
            if: always()
            with:
              name: evalgate-results
              path: .evalgate/
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk check — platform gate">
    Runs the regression gate against the EvalGate platform (requires `EVALGATE_API_KEY`). Use `--onFail import` to upload failed run context to the dashboard for review.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk check --format github --onFail import
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk baseline update — refresh the baseline">
    Re-runs your tests and overwrites the stored baseline with the new results. Run this after you intentionally change model behavior or fix a known issue.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk baseline update
    ```
  </Accordion>
</AccordionGroup>

## Labeling and analysis

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk label — interactive trace labeling">
    Steps through your unlabeled traces one by one. Use arrow keys to select pass/fail and pick a failure mode. Press `u` to undo the previous label. Press `Ctrl-C` to save progress and exit.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk label
    ```

    Each label you save becomes part of the golden dataset that can be used by later eval runs and gates.
  </Accordion>

  <Accordion title="npx @evalgate/sdk analyze — failure-mode frequency report">
    Aggregates labeled traces and prints a frequency report of failure modes (counts and share of labeled failures).

    ```bash theme={null} theme={null}
    npx @evalgate/sdk analyze
    npx @evalgate/sdk analyze --format json
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk failure-modes — configure failure-mode taxonomy">
    Lists standard failure modes used by `label`, and optionally seeds or extends your `evalgate.config.json` alert weights.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk failure-modes
    npx @evalgate/sdk failure-modes --init
    npx @evalgate/sdk failure-modes --add my_custom_mode
    npx @evalgate/sdk failure-modes --json
    ```

    Custom modes are stored under `failureModeAlerts.modes` in config and used by `analyze` and gate alerting.
  </Accordion>

  <Accordion title="npx @evalgate/sdk replay-candidate — inspect a stored candidate for replay">
    Loads a candidate eval case from the platform and prints the minimized input for manual or scripted re-run. The legacy `replay` alias is deprecated — prefer `replay-candidate`.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk replay-candidate <candidate-id>
    npx @evalgate/sdk replay-candidate 42 --format json --model gpt-5.2-chat-latest
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk replay-decision — compare two runs">
    Loads two saved run artifacts and emits a keep/discard decision for each case — useful for reviewing whether a prompt change improved or regressed specific failure modes.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk replay-decision \
      --previous .evalgate/runs/run-prev.json \
      --current  .evalgate/runs/run-latest.json
    ```
  </Accordion>
</AccordionGroup>

## Agent evidence and enterprise controls

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk run-cli — run a command with evidence capture">
    Wraps an agent or CLI subprocess, captures stdout/stderr and tool events, and posts an evidence envelope to EvalGate (unless `--local`).

    ```bash theme={null} theme={null}
    npx @evalgate/sdk run-cli \
      --runtime support-agent \
      --permission-profile default \
      -- npm test -- --grep "billing"
    ```

    Useful flags:

    ```bash theme={null} theme={null}
    npx @evalgate/sdk run-cli --runtime my-runtime -- -- my-agent.sh
    npx @evalgate/sdk run-cli --local --format json -- my-agent.sh
    npx @evalgate/sdk run-cli --strict-persistence --evidence-status .evalgate/run-cli-status.json -- my-agent.sh
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk mcp — inspect, audit, or proxy MCP servers">
    Validates MCP manifests, audits connector readiness, or proxies a server while emitting evidence.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk mcp inspect --manifest ./mcp/manifest.json
    npx @evalgate/sdk mcp audit --connector filesystem --manifest ./mcp/manifest.json
    npx @evalgate/sdk mcp proxy --runtime mcp-host --server filesystem -- npx @modelcontextprotocol/server-filesystem /tmp
    ```

    Add `--format json` for machine-readable audit output. Use `--strict-persistence` in CI when evidence must land on the platform.
  </Accordion>

  <Accordion title="npx @evalgate/sdk controls — inspect enterprise guardrails">
    Fetches org-scoped system controls: provider allowlists, PII scrub mode, retention policy, drift alerts, and guardrail state.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk controls
    npx @evalgate/sdk controls --format json
    ```

    Requires `EVALGATE_API_KEY` with access to `/api/system/controls`.
  </Accordion>

  <Accordion title="npx @evalgate/sdk share — create a share link for a run">
    Creates a time-limited share link for an evaluation run. Use `--format json` to emit the token and URL for automation.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk share \
      --evaluationId 42 \
      --runId 9001 \
      --expires 7d \
      --format json
    ```
  </Accordion>
</AccordionGroup>

## Repository intelligence and evaluation packs

Use the local activation path first. After the gate is working and GitHub is
connected, scan an exact commit, review the evidence, and then install coverage:

```bash theme={null} theme={null}
npx @evalgate/sdk repo repositories
npx @evalgate/sdk repo scan --repository 42 --head-sha <40-character-sha>
npx @evalgate/sdk repo ask --repository 42 \
  --question "What AI models, agents, tools, and evals exist?"
npx @evalgate/sdk packs list --domain coding
npx @evalgate/sdk packs install coding-agent-release-safety
```

Repository scanning never executes repository code. Source findings and pack
recommendations remain evidence-bounded and report incomplete scans honestly.

## Advanced

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk cluster — group similar failures">
    Reads a saved run artifact and groups cases with similar failure patterns. Use the output to prioritize which failure mode to fix first.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk cluster --run .evalgate/runs/latest.json
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk synthesize — generate synthetic golden cases">
    Reads your labeled failure dataset and generates deterministic synthetic test cases to expand coverage of underrepresented failure modes.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk synthesize \
      --dataset .evalgate/golden/labeled.jsonl \
      --output  .evalgate/golden/synthetic.jsonl
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk auto — bounded autonomous prompt-improvement loop">
    Reads labeled failures and prior prompt history, generates the next candidate prompt edit, evaluates it against impacted specs, and keeps the edit only if it does not regress any existing case. The loop terminates on explicit guard conditions rather than running open-ended.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk auto \
      --objective tone_mismatch \
      --prompt prompts/support.md \
      --autonomous \
      --budget 3
    ```

    To repeat bounded cycles unattended (for example, overnight):

    ```bash theme={null} theme={null}
    npx @evalgate/sdk auto daemon --cycles 5
    ```

    <Note>
      The TypeScript CLI uses `npx @evalgate/sdk auto`. The Python CLI exposes the same bounded workflow as `evalgate auto run` and `evalgate auto daemon`.
    </Note>
  </Accordion>

  <Accordion title="npx @evalgate/sdk discover --manifest — refresh the spec manifest">
    Scans your project for eval spec files, refreshes the manifest, and reports any redundant or overlapping specs.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk discover --manifest
    ```
  </Accordion>
</AccordionGroup>

## Judge commands

<AccordionGroup>
  <Accordion title="npx @evalgate/sdk judge registry — list available judges">
    Prints all judges available in the EvalGate registry for your organization.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk judge registry
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk judge presets — list judge presets">
    Prints the built-in judge presets (pre-configured provider + model + prompt combinations).

    ```bash theme={null} theme={null}
    npx @evalgate/sdk judge presets
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk judge test — test a judge configuration">
    Runs a judge against a single input/output pair and prints the score, reasoning, and signals. Use this to validate a judge configuration before wiring it into your gate.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk judge test \
      --provider openai \
      --model gpt-5.2-chat-latest \
      --judge support_quality \
      --input "Cancel my subscription" \
      --output "I've canceled your plan effective today."
    ```

    Example output:

    ```json theme={null} theme={null}
    {
      "score": 0.92,
      "passed": true,
      "reasoning": "The response directly addresses the user's request with a clear confirmation.",
      "signals": ["direct", "action_confirmed", "professional_tone"]
    }
    ```
  </Accordion>

  <Accordion title="npx @evalgate/sdk judge compare — compare two outputs">
    Runs a judge against two candidate outputs for the same input and returns a preference decision with reasoning. Useful for A/B prompt comparisons.

    ```bash theme={null} theme={null}
    npx @evalgate/sdk judge compare \
      --config-id 42 \
      --input "Cancel my subscription" \
      --output-a "I've canceled your plan effective today." \
      --output-b "Please visit billing settings to make changes."
    ```
  </Accordion>
</AccordionGroup>

## Judge credibility config

Configure judge credibility thresholds and failure-mode alerts in `evalgate.config.json` at the root of your project:

```json theme={null} theme={null}
{
  "judge": {
    "bootstrapSeed": 42,
    "tprMin": 0.70,
    "tnrMin": 0.70,
    "minLabeledSamples": 30
  },
  "failureModeAlerts": {
    "modes": {
      "hallucination": { "weight": 1.5, "maxPercent": 10 },
      "off_topic":     { "weight": 1.0, "maxPercent": 20, "maxCount": 5 },
      "wrong_format":  { "weight": 0.8, "maxPercent": 15 }
    }
  }
}
```

<Tip>
  Set `bootstrapSeed` to a fixed value (for example, `42`) to make judge credibility calculations deterministic across CI runs. Without a fixed seed, bootstrap confidence intervals may vary slightly between runs.
</Tip>

When a judge's discriminative power (TPR + TNR − 1) falls at or below 0.05, the gate skips score correction and exits with code `8` (WARN) instead of using a potentially biased score. When labeled sample count is below `minLabeledSamples`, bootstrap confidence intervals are also skipped — both conditions emit reason codes into the `judgeCredibility` block of the JSON report.

## Complete command index

Every registered EvalGate CLI command. Run `npx @evalgate/sdk <command> --help` for the same text in your terminal.

<AccordionGroup>
  <Accordion title="evalgate analyze — Analyze labeled golden dataset failure modes (first pass)">
    ```text theme={null} theme={null}
    evalgate analyze — Analyze labeled golden dataset failure modes (first pass)

    Usage:
    evalgate analyze [options]

    Options:
    --dataset <path>  Labeled JSONL dataset path (default: .evalgate/golden/labeled.jsonl)
    --format <fmt>    Output format: human (default), json
    --top <n>         Number of top failure modes to show (default: 5)
    ```
  </Accordion>

  <Accordion title="evalgate api - call any public operation from the verified OpenAPI contract">
    ```text theme={null} theme={null}
    evalgate api - call any public operation from the verified OpenAPI contract

    Usage:
    evalgate api <operation-id> [options]
    evalgate api --list-operations [--json]

    Options:
    --path <name=value>        Path parameter; repeat for multiple values
    --query <name=value>       Query parameter; repeat for multiple values
    --header <name=value>      Declared operation header; repeat as needed
    --body <json|@file>        JSON request body or @path to a JSON file
    --idempotency-key <key>    Send Idempotency-Key for a mutation
    --paginate                 Fetch every declared cursor/offset/page response
    --limit <n>                Page size when the operation declares limit
    --max-pages <n>            Pagination safety bound (default: 100)
    --timeout-ms <n>           Request timeout in milliseconds
    --api-key <key>            API key (or EVALGATE_API_KEY)
    --base-url <url>           API base URL (or EVALGATE_BASE_URL)
    --organization-id <uuid>  Organization context (or EVALGATE_ORGANIZATION_ID)
    --json                     Compact machine-readable JSON on stdout

    Exit codes: 0 success, 4 API/network, 5 invalid arguments, 11 local IO, 99 internal, 130 cancelled.
    ```
  </Accordion>

  <Accordion title="evalgate auto — Plan, run, or daemonize budget-aware experiment iterations">
    ```text theme={null} theme={null}
    evalgate auto — Plan, run, or daemonize budget-aware experiment iterations

    Usage:
    evalgate auto [options]
    evalgate auto run [options]
    evalgate auto daemon [options]
    evalgate auto history [options]

    Options:
    --objective <text>          Target failure mode or experiment goal (required)
    --hypothesis <text>         Human-readable candidate hypothesis
    --base <ref>                Baseline run report reference (default: baseline)
    --head <path>               Candidate run report path to evaluate
    --budget <n>                Planned iteration budget (default: 3)
    --dry-run                   Produce a plan without making a keep/discard decision
    --output <path>             Auto report JSON path (default: .evalgate/auto/latest.json)
    --format <fmt>              Output format: human (default), json
    --cycles <n>                Daemon mode cycle count (daemon subcommand)
    --interval-ms <ms>          Daemon mode wait interval between cycles
    ```
  </Accordion>

  <Accordion title="evalgate baseline — Manage regression gate baselines">
    ```text theme={null} theme={null}
    evalgate baseline — Manage regression gate baselines

    Usage:
    evalgate baseline init     Create starter evals/baseline.json
    evalgate baseline update   Run tests and update baseline
    ```
  </Accordion>

  <Accordion title="evalgate capabilities - Navigate EvalGate by product outcome">
    ```text theme={null} theme={null}
    evalgate capabilities - Navigate EvalGate by product outcome

    Usage:
    evalgate capabilities
    evalgate capabilities <goal>
    evalgate capabilities [goal] --format json

    Options:
    --format <fmt>  Output format: human (default), json
    --json          Alias for --format json

    The versioned capability contract maps each governed workflow to its web route,
    native TypeScript/Python commands, public operation IDs, artifacts, and next step.
    ```
  </Accordion>

  <Accordion title="evalgate check — CI/CD evaluation gate (API-based)">
    ```text theme={null} theme={null}
    evalgate check — CI/CD evaluation gate (API-based)

    Usage:
    evalgate check [options]

    Options:
    --evaluationId <id>  Evaluation to gate on
    --apiKey <key>       API key (or EVALGATE_API_KEY env)
    --format <fmt>       Output format: human (default), json, github
    --explain            Show score breakdown
    --minScore <n>       Fail if score < n
    --maxDrop <n>        Fail if score dropped > n
    --policy <name>      Enforce policy (HIPAA, SOC2, etc.)

    Examples:
    evalgate check --minScore 92 --evaluationId 42
    ```
  </Accordion>

  <Accordion title="evalgate ci — One-command CI loop (manifest → impact → run → diff)">
    ```text theme={null} theme={null}
    evalgate ci — One-command CI loop (manifest → impact → run → diff)

    Usage:
    evalgate ci [options]

    Options:
    --base <ref>       Base reference for diff
    --impacted-only    Run only impacted specs
    --format <fmt>     Output format: human (default), json, github
    --write-results    Write run results
    ```
  </Accordion>

  <Accordion title="evalgate cluster — Group similar traces for faster cluster-level review">
    ```text theme={null} theme={null}
    evalgate cluster — Group similar traces for faster cluster-level review

    Usage:
    evalgate cluster [options]

    Options:
    --run <path>            Run result JSON to cluster (default: searches latest run)
    --clusters <n>          Requested number of clusters (default: auto)
    --include-passed        Include passing traces instead of only failures
    --output <path>         Write cluster report JSON to disk
    --format <fmt>          Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate compare — Side-by-side result file comparison">
    ```text theme={null} theme={null}
    evalgate compare — Side-by-side result file comparison

    Compares two or more saved run result JSON files. Does NOT re-run anything.
    You run each model/config separately (evalgate run --write-results), then compare the artifacts.

    Usage:
    evalgate compare --base <file> --head <file> [options]
    evalgate compare --runs <file1> <file2> [file3...] [options]

    Options:
    --base <file>      Baseline run result JSON file
    --head <file>      Head run result JSON file
    --runs <files>     N-way compare (3+ run result JSON files)
    --labels <names>   Optional cosmetic labels for the output table (e.g., model names)
    --format <fmt>     Output format: human (default), json
    --sort-by <key>    Sort by: name (default), score, duration

    Examples:
    evalgate compare --base .evalgate/runs/run-a.json --head .evalgate/runs/run-b.json
    evalgate compare --base gpt4o.json --head claude.json --labels "GPT-4o" "Claude 3.5"
    evalgate compare --runs run-a.json run-b.json run-c.json
    ```
  </Accordion>

  <Accordion title="evalgate controls - Inspect enterprise guardrails, provider policy, retention, and drift alerts">
    ```text theme={null} theme={null}
    evalgate controls - Inspect enterprise guardrails, provider policy, retention, and drift alerts

    Usage:
    evalgate controls [options]

    Options:
    --apiKey <key>      API key (or EVALGATE_API_KEY env)
    --baseUrl <url>     API base URL
    --format <fmt>      Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate datasets - Manage governed Dataset Hub artifacts">
    ```text theme={null} theme={null}
    evalgate datasets - Manage governed Dataset Hub artifacts

    Usage:
    evalgate datasets list [--status active]
    evalgate datasets create --name <name> [--slug <slug>]
    evalgate datasets import --dataset <id> --file <path> [--import-format jsonl]
    evalgate datasets version --dataset <id>
    evalgate datasets publish --dataset <id> --version <id>
    evalgate datasets bind --evaluation <id> --dataset <id> --version <id> [--mode snapshot]
    evalgate datasets bindings --evaluation <id>

    Options:
    --idempotency-key <key>  Stable key for safe retries; generated when omitted
    --apiKey <key>           API key (or EVALGATE_API_KEY)
    --baseUrl <url>          API base URL
    --format <fmt>           human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate diff — Compare two run reports">
    ```text theme={null} theme={null}
    evalgate diff — Compare two run reports

    Usage:
    evalgate diff [options]

    Options:
    --base <ref>   Base branch or report path
    --head <path>  Head report path
    --format <fmt> Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate discover — Discover behavioral specs">
    ```text theme={null} theme={null}
    evalgate discover — Discover behavioral specs

    Usage:
    evalgate discover [options]

    Options:
    --manifest  Generate evaluation manifest for incremental analysis

    Also reports suite diversity and potentially redundant spec pairs.
    ```
  </Accordion>

  <Accordion title="evalgate doctor — Comprehensive CI/CD readiness checklist">
    ```text theme={null} theme={null}
    evalgate doctor — Comprehensive CI/CD readiness checklist

    Usage:
    evalgate doctor [options]

    Options:
    --report         Output JSON diagnostic bundle
    --quick           Skip live API checks (local-only mode)
    --format <fmt>    Output format: human (default), json
    --strict          Treat warnings as failures

    Runs itemized pass/fail checks with exact remediation commands.
    ```
  </Accordion>

  <Accordion title="evalgate evals — Evals-as-Code manifest workflow">
    ```text theme={null} theme={null}
    evalgate evals — Evals-as-Code manifest workflow

    Usage:
    evalgate evals validate -f evalgate.yml
    evalgate evals plan -f evalgate.yml
    evalgate evals apply -f evalgate.yml
    evalgate evals diff -f evalgate.yml
    evalgate evals gate -f evalgate.yml

    Options:
    -f, --file <path>       Manifest path (default: evalgate.yml)
    --format <fmt>          human (default) or json
    --apiKey <key>          API key (or EVALGATE_API_KEY)
    --baseUrl <url>         API base URL
    --evaluationId <id>     Gate: evaluation id in manifest
    --runId <n>             Gate: optional run id
    ```
  </Accordion>

  <Accordion title="evalgate explain — Explain last gate/check failure">
    ```text theme={null} theme={null}
    evalgate explain — Explain last gate/check failure

    Usage:
    evalgate explain [options]

    Options:
    --report <path>  Path to report JSON (default: evals/regression-report.json)
    --format <fmt>   Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate failure-modes — View and configure failure-mode taxonomy">
    ```text theme={null} theme={null}
    evalgate failure-modes — View and configure failure-mode taxonomy

    Usage:
    evalgate failure-modes [options]

    Options:
    --init            Seed standard failure modes into evalgate.config.json
    --add <name>      Add a custom failure mode to config alerts
    --json            Machine-readable output
    ```
  </Accordion>

  <Accordion title="evalgate gate — Run the regression gate">
    ```text theme={null} theme={null}
    evalgate gate — Run the regression gate

    Usage:
    evalgate gate [options]

    Options:
    --format <fmt>   Output format: human (default), json, github
    --dry-run        Run checks but always exit 0 (preview mode)

    Examples:
    evalgate gate
    evalgate gate --format json
    evalgate gate --dry-run
    ```
  </Accordion>

  <Accordion title="evalgate generate - Generate test cases via LLM">
    ```text theme={null} theme={null}
    evalgate generate - Generate test cases via LLM

    Usage:
    evalgate generate --scenario <text> [options]

    Options:
    --scenario <text>    Scenario description (required)
    --count <n>          Number of cases to generate (default: 5)
    --provider <name>    LLM provider: openai or anthropic (default: openai)
    --model <name>       Model name (default: gpt-4o-mini)
    --output <path>      Output JSONL path (default: .evalgate/golden/generated.jsonl)
    --format <fmt>       Output format: human (default), json

    Environment:
    OPENAI_API_KEY       Required when --provider openai
    ANTHROPIC_API_KEY    Required when --provider anthropic

    Uses an LLM to generate diverse, realistic test cases from a scenario
    description. Cases are output as JSONL, ready for review.
    ```
  </Accordion>

  <Accordion title="evalgate impact-analysis — Analyze impact of changes">
    ```text theme={null} theme={null}
    evalgate impact-analysis — Analyze impact of changes

    Usage:
    evalgate impact-analysis [options]

    Options:
    --base <branch>          Base branch (default: main)
    --changed-files <files>  Comma-separated list of changed files
    --format <fmt>           Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate init - Preview or apply EvalGate project scaffolding">
    ```text theme={null} theme={null}
    evalgate init - Preview or apply EvalGate project scaffolding

    Usage:
    evalgate init [options]

    Options:
    --apply            Write the displayed scaffold; default is a non-mutating plan
    --template <name>  Start with a real working template (chatbot, codegen, agent, safety, rag, production-loop)
    --list-templates   Show all available templates

    Default behavior is safe: it writes no files and executes no project commands.
    ```
  </Accordion>

  <Accordion title="evalgate integrity - Inspect integrity and workflow-governance signals">
    ```text theme={null} theme={null}
    evalgate integrity - Inspect integrity and workflow-governance signals

    Usage:
    evalgate integrity --evaluationId <id> [options]

    Options:
    --run <id>          Optional run ID filter
    --refresh           Recompute the snapshot before printing
    --apiKey <key>      API key (or EVALGATE_API_KEY env)
    --baseUrl <url>     API base URL
    --format <fmt>      Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate judge — Run, compare, or configure pluggable judges">
    ```text theme={null} theme={null}
    evalgate judge — Run, compare, or configure pluggable judges

    Usage:
    evalgate judge configs [options]
    evalgate judge create --name <name> --promptTemplate <text> [--judge provider:model ...] [options]
    evalgate judge evaluate --configId <id> --input <text> --output <text> [options]
    evalgate judge compare --configId <id> --input <text> --outputA <text> --outputB <text> [options]
    evalgate judge test --promptTemplate <text> --input <text> --output <text> [--judge provider:model ...] [options]

    Options:
    --judge <ref>          Repeatable. Use provider:model or rule:exact_match
    --aggregation <mode>   all_pass, any_pass, weighted, primary_fallback
    --provider <name>      openai, anthropic, google, local
    --model <name>         Primary model when --judge is omitted
    --apiKey <key>         API key
    --baseUrl <url>        API base URL
    --format <fmt>         Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate knowledge - Inspect learned workflow hints and playbook guidance">
    ```text theme={null} theme={null}
    evalgate knowledge - Inspect learned workflow hints and playbook guidance

    Usage:
    evalgate knowledge --evaluationId <id> [options]

    Options:
    --failure-mode <m>   Restrict to one failure mode
    --workflow-class <c> Restrict to one workflow class
    --apiKey <key>       API key (or EVALGATE_API_KEY env)
    --baseUrl <url>      API base URL
    --format <fmt>       Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate label — Interactive trace labeling for golden dataset">
    ```text theme={null} theme={null}
    evalgate label — Interactive trace labeling for golden dataset

    Usage:
    evalgate label [options]

    Options:
    --run <path>       Run result JSON to label (default: searches evals/latest-run.json)
    --cluster <path>   Cluster report JSON to label in cluster order
    --output <path>    Labeled JSONL output path (default: .evalgate/golden/labeled.jsonl)
    --format <fmt>     Output format: human (default), json

    Steps through traces from a run result or cluster report, allowing pass/fail labeling
    and optional failure-mode tagging. Writes to canonical labeled.jsonl.
    ```
  </Accordion>

  <Accordion title="evalgate mcp - Inspect, audit, or proxy MCP servers with evidence capture">
    ```text theme={null} theme={null}
    evalgate mcp - Inspect, audit, or proxy MCP servers with evidence capture

    Usage:
    evalgate mcp inspect --manifest <manifest.json>
    evalgate mcp audit --connector <key> --manifest <manifest.json>
    evalgate mcp proxy --runtime <key> --server <key> -- <server-command> [args...]

    Options:
    --local                         Do not post evidence to EvalGate
    --idempotency-key <key>         Stable key for proxy session retries
    --evidence-status <path>        Write machine-readable audit/proxy status JSON
    --strict-persistence            Exit nonzero if evidence persistence fails or conflicts
    --apiKey <key>                  API key (or EVALGATE_API_KEY env)
    --baseUrl <url>                 API base URL
    --format <fmt>                  Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate measurement - Inspect and execute evaluator validation">
    ```text theme={null} theme={null}
    evalgate measurement - Inspect and execute evaluator validation

    Usage:
    evalgate measurement runs [--status completed]
    evalgate measurement releases
    evalgate measurement show --id <validation-run-id>
    evalgate measurement metrics --id <validation-run-id>
    evalgate measurement execute --id <validation-run-id>

    Options:
    --limit <n>       Result limit (1-100)
    --apiKey <key>    API key (or EVALGATE_API_KEY)
    --baseUrl <url>   API base URL
    --format <fmt>    human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate migrate — Migrate legacy config formats">
    ```text theme={null} theme={null}
    evalgate migrate — Migrate legacy config formats

    Usage:
    evalgate migrate config --in <input> --out <output> [options]

    Options:
    --in, -i <path>       Input config path
    --out, -o <path>      Output DSL path
    --verbose, -v         Verbose migration logs
    --no-helpers          Omit helper specs from output
    --no-preserve-ids     Do not preserve spec IDs
    --no-provenance       Omit provenance metadata
    --dry-run             Print to stdout instead of writing file
    ```
  </Accordion>

  <Accordion title="evalgate packs - Discover and install first-party domain evaluation packs">
    ```text theme={null} theme={null}
    evalgate packs - Discover and install first-party domain evaluation packs

    Usage:
    evalgate packs list [--domain healthcare]
    evalgate packs show <pack-id>
    evalgate packs install <pack-id> [--name <name>]
    evalgate packs installations
    evalgate packs installation <installation-id>

    Options:
    --idempotency-key <key>  Stable key for safe installation retries
    --apiKey <key>           API key (or EVALGATE_API_KEY)
    --baseUrl <url>          API base URL
    --format <fmt>           human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate plan - Preview an autonomous workflow plan">
    ```text theme={null} theme={null}
    evalgate plan - Preview an autonomous workflow plan

    Usage:
    evalgate plan --objective <text> --target-path <path> --strategy-tracks <ids> [options]

    Options:
    --target-file <path>      Read target content from disk instead of inline text
    --target-content <text>   Inline target content
    --iteration <n>           Planner iteration (default: 1)
    --hypothesis <text>       Optional experiment hypothesis
    --workflow-class <cls>    Optional workflow class override
    --autonomy-level <lvl>    Optional autonomy override
    --approval-mode <mode>    Optional approval mode override
    --memory-mode <mode>      Optional memory mode override
    --background-execution    Mark the plan as background-capable
    --required-hooks <list>   Comma-separated required hooks
    --apiKey <key>            API key (or EVALGATE_API_KEY env)
    --baseUrl <url>           API base URL
    --format <fmt>            Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate playground - Author governed Playground cases">
    ```text theme={null} theme={null}
    evalgate playground - Author governed Playground cases

    Usage:
    evalgate playground import --evaluation <id> --file <path> [--import-format jsonl]
    evalgate playground generate --evaluation <id> --scenario <text> [--count 5]
    evalgate playground from-trace --evaluation <id> --trace <id> [--failure-span <id>]

    Options:
    --apiKey <key>           API key (or EVALGATE_API_KEY)
    --baseUrl <url>          API base URL
    --format <fmt>           human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate print-config — Show resolved config">
    ```text theme={null} theme={null}
    evalgate print-config — Show resolved config

    Usage:
    evalgate print-config [options]

    Options:
    --format <fmt>  Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate promote — Promote candidate eval cases to regression suite">
    ```text theme={null} theme={null}
    evalgate promote — Promote candidate eval cases to regression suite

    Usage:
    evalgate promote <candidate-id>     Promote a specific candidate
    evalgate promote --auto             Auto-promote all eligible
    evalgate promote --list             List promotable candidates

    Options:
    --evaluation-id <id>  Target evaluation (default: golden regression)
    --apiKey <key>        API key
    --baseUrl <url>       API base URL
    --dry-run             Show what would be promoted without making changes
    ```
  </Accordion>

  <Accordion title="evalgate prompts - Manage governed Prompt Hub artifacts">
    ```text theme={null} theme={null}
    evalgate prompts - Manage governed Prompt Hub artifacts

    Usage:
    evalgate prompts list [--status active]
    evalgate prompts create --key <key> --name <name> --definition <json-file>
    evalgate prompts show --prompt <id>
    evalgate prompts version --prompt <id> --definition <json-file>
    evalgate prompts review --prompt <id> --version <id> --revision <n>
    evalgate prompts approve --prompt <id> --version <id> --revision <n>
    evalgate prompts publish --prompt <id> --version <id> --environment <name> --binding-revision <n>
    evalgate prompts resolve --prompt <id> --environment <name>
    evalgate prompts rollback --prompt <id> --environment <name> --binding-version <id> --binding-revision <n>

    Options:
    --idempotency-key <key>  Stable key for safe retries; generated when omitted
    --apiKey <key>           API key (or EVALGATE_API_KEY)
    --baseUrl <url>          API base URL
    --format <fmt>           human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate replay — Replay a candidate eval case (deprecated alias)">
    ```text theme={null} theme={null}
    evalgate replay — Replay a candidate eval case (deprecated alias)

    Deprecated: use \
    ```
  </Accordion>

  <Accordion title="evalgate replay-candidate — Replay a candidate eval case">
    ```text theme={null} theme={null}
    evalgate replay-candidate — Replay a candidate eval case

    Usage:
    evalgate replay-candidate <candidate-id>

    Options:
    --model <model>   Override model
    --format <fmt>    Output format: human (default), json
    --apiKey <key>    API key
    --baseUrl <url>   API base URL
    ```
  </Accordion>

  <Accordion title="evalgate replay-decision - Compare two run reports and decide keep/discard">
    ```text theme={null} theme={null}
    evalgate replay-decision - Compare two run reports and decide keep/discard

    Usage:
    evalgate replay-decision --previous <run.json|latest> --current <run.json> [options]

    Options:
    --format <fmt>    Output format: human (default), json

    Requires normalizedBudget in evalgate.config.json.
    ```
  </Accordion>

  <Accordion title="evalgate repo - Scan an exact connected repository commit for AI systems">
    ```text theme={null} theme={null}
    evalgate repo - Scan an exact connected repository commit for AI systems

    Usage:
    evalgate repo repositories
    evalgate repo scan --repository <id> [--head-sha <sha>]
    evalgate repo scans --repository <id>
    evalgate repo show --repository <id> --scan <scan-id>
    evalgate repo ask --repository <id> --question <text>

    Options:
    --idempotency-key <key>  Stable key for safe scan retries
    --apiKey <key>           API key (or EVALGATE_API_KEY)
    --baseUrl <url>          API base URL
    --format <fmt>           human (default) or json
    ```
  </Accordion>

  <Accordion title="evalgate rollback - Remove files created by one init transaction">
    ```text theme={null} theme={null}
    evalgate rollback - Remove files created by one init transaction

    Usage:
    evalgate rollback --transaction <id>
    evalgate rollback <id>

    Options:
    --transaction <id>  Transaction recorded in evalgate.project.json

    Rollback removes only transaction-owned files that have not been edited since
    init. Pre-existing and subsequently edited files are preserved.
    ```
  </Accordion>

  <Accordion title="evalgate run — Run evaluation specifications">
    ```text theme={null} theme={null}
    evalgate run — Run evaluation specifications

    Usage:
    evalgate run [options]

    Options:
    --spec-ids <ids>    Comma-separated list of spec IDs
    --impacted-only     Run only impacted specs (requires --base)
    --base <branch>     Base branch for impact analysis
    --format <fmt>      Output format: human (default), json
    --write-results     Write results to .evalgate/last-run.json
    ```
  </Accordion>

  <Accordion title="evalgate run-cli - Run an agent/CLI command with evidence capture">
    ```text theme={null} theme={null}
    evalgate run-cli - Run an agent/CLI command with evidence capture

    Usage:
    evalgate run-cli --runtime <key> [--permission-profile <key>] -- <command> [args...]

    Options:
    --runtime <key>                 Agent runtime key
    --permission-profile <key>      Permission profile key
    --idempotency-key <key>         Stable key for safe retries
    --evidence-status <path>        Write machine-readable persistence status JSON
    --local                         Capture locally without posting to EvalGate
    --strict-persistence            Exit nonzero if EvalGate persistence fails or conflicts
    --apiKey <key>                  API key (or EVALGATE_API_KEY env)
    --baseUrl <url>                 API base URL
    --format <fmt>                  Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate share — Create share link for a run">
    ```text theme={null} theme={null}
    evalgate share — Create share link for a run

    Usage:
    evalgate share [options]

    Options:
    --scope <s>         Share scope
    --evaluationId <id> Evaluation ID
    --runId <id>        Run ID
    --expires <dur>     Expiry duration (e.g. 7d)
    --apiKey <key>      API key
    --format <fmt>      Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate sync — Push/pull labels and prompt state between CLI and web">
    ```text theme={null} theme={null}
    evalgate sync — Push/pull labels and prompt state between CLI and web

    Usage:
    evalgate sync [options]

    Options:
    --project-key <key>       Project key slug (from evalgate.config.json or --project-key)
    --evaluation-id <id>      Direct evaluation ID (bypasses project-key resolution)
    --prompt-id <id>          Prompt ID to pull or push prompt versions for
    --target-path <path>      Local path for prompt pull/push
    --labeled-dataset <path>  JSONL file of labeled cases to push or merge into
    --pull-labels             Pull web-labeled cases into the local JSONL file
    --push-prompt             Push target-path content as a web prompt version
    --pull-prompt             Pull active prompt content when also pushing
    --expected-prompt-active-version-id <id>  Expected active prompt version for stale-write protection
    --dry-run                 Show what would happen without making changes
    --format <fmt>            Output format: human (default), json

    Examples:
    evalgate sync --project-key my-chatbot
    evalgate sync --project-key my-chatbot --labeled-dataset .evalgate/golden/labeled.jsonl --pull-labels
    evalgate sync --evaluation-id 42 --prompt-id prompt_1 --target-path prompts/system.md --push-prompt
    evalgate sync --dry-run
    ```
  </Accordion>

  <Accordion title="evalgate synthesize — Generate synthetic golden-case drafts from labeled failures">
    ```text theme={null} theme={null}
    evalgate synthesize — Generate synthetic golden-case drafts from labeled failures

    Usage:
    evalgate synthesize [options]

    Options:
    --dataset <path>            Labeled JSONL dataset path (default: .evalgate/golden/labeled.jsonl)
    --dimensions <path>         Dimension matrix JSON path
    --failure-mode <name>       Restrict to one or more failure modes (comma-separated)
    --count <n>                 Number of synthetic cases to generate (default: mode × dimension coverage)
    --output <path>             Synthetic JSONL output path (default: .evalgate/golden/synthetic.jsonl)
    --format <fmt>              Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate trace — Inspect stored trajectory metrics for a run">
    ```text theme={null} theme={null}
    evalgate trace — Inspect stored trajectory metrics for a run

    Usage:
    evalgate trace --evaluationId <id> --run <id> [options]

    Options:
    --apiKey <key>      API key (or EVALGATE_API_KEY env)
    --baseUrl <url>     API base URL
    --format <fmt>      Output format: human (default), json
    --case <id>         Restrict output to one test case
    ```
  </Accordion>

  <Accordion title="evalgate validate — Validate spec files without running them">
    ```text theme={null} theme={null}
    evalgate validate — Validate spec files without running them

    Usage:
    evalgate validate [options]

    Options:
    --format <fmt>  Output format: human (default), json
    ```
  </Accordion>

  <Accordion title="evalgate watch — Watch mode (re-execute on file save)">
    ```text theme={null} theme={null}
    evalgate watch — Watch mode (re-execute on file save)

    Usage:
    evalgate run --watch [options]
    evalgate watch [options]

    Options:
    --debounce <ms>    Debounce interval (default: 300ms)
    --no-clear         Don't clear screen between runs
    --format <fmt>     Output format: human (default), json
    --write-results    Write results to .evalgate/last-run.json

    Examples:
    evalgate run --watch
    evalgate watch --write-results
    ```
  </Accordion>

  <Accordion title="evalgate workflow - Inspect or generate workflow-native coverage bundles">
    ```text theme={null} theme={null}
    evalgate workflow - Inspect or generate workflow-native coverage bundles

    Usage:
    evalgate workflow --evaluationId <id> [options]

    Options:
    --synthesize           Generate workflow cases and experiment recipes
    --run <id>             Optional source run for synthesis
    --case-count <n>       Optional generated case count (max 20)
    --include-passed       Include passed runs when synthesizing
    --persist-candidates   Persist generated candidate cases
    --persist-artifacts    Persist workflow artifacts
    --apiKey <key>         API key (or EVALGATE_API_KEY env)
    --baseUrl <url>        API base URL
    --format <fmt>         Output format: human (default), json
    ```
  </Accordion>
</AccordionGroup>