Skip to main content

​
EvalGate CLI command reference

Complete reference for all EvalGate CLI commands: setup, gates, CI integration, trace labeling, failure analysis, judge orchestration, and auto loops.
The EvalGate CLI is the fastest way to run regression gates, analyze failure patterns, and automate prompt improvement without leaving your terminal. Run TypeScript CLI commands with npx @evalgate/sdk <command> for zero-install usage, or add @evalgate/sdk to your project and run the same commands through your package manager.
Both CLIs expose the same versioned capability map. Start here when you know the job you need to accomplish but not the internal command name:
The complete map of governed core workflows covers setup, evaluation, Playground, Prompt Hub, Dataset Hub, first-party evaluation packs, repository intelligence, experiments, scorer development, logs, continuous evaluation, failure topics, dashboards, Copilot, provider onboarding, red-team testing, remote runners, deployable assets, governance, and release. TypeScript and Python emit identical JSON from the generated contract. Every mapped workflow names its web route, public operation IDs, native automation command, artifacts, and next governed step; generation fails when any required workflow disappears.

​
Stable exit codes

​
Call any public API operation

The TypeScript CLI’s api command consumes the generated public operation registry. This gives automation a single command for every operation in the API reference without falling back to private routes or database helpers. Treat it as an escape hatch for advanced automation; evalgate capabilities and the native workflow commands are the primary navigation surface.
Use --path name=value, --query name=value, and --header name=value more than once when an operation declares multiple inputs. --body accepts inline JSON or @path/to/body.json. Machine mode writes the exact successful API envelope to stdout; failures write a stable JSON error containing code, message, status, requestId, and exitCode. Ctrl-C aborts the active request and exits 130.

​
Setup and initialization

Detects Node, Python, polyglot, and nested workspace projects and defaults to a non-mutating plan. Review the files, commands, network boundary, and rollback instructions, then apply explicitly. Init creates a pending baseline without executing tests or accepting results, and it requires no account.
Supported first-class handlers are npm, pnpm, the legacy yarn alias, Yarn Classic, Yarn Modern, Bun, Deno, pip, uv, Poetry, Pipenv, PDM, Conda, Mamba, Hatch, Pixi, and explicit custom JSON argv. Conflicting JavaScript or Python/environment markers stop the plan. Repeat --package-handler <id> for an explicit polyglot composition; gate and baseline update execute every runnable handler from its owning root.
baseline update accepts only successful current evidence. Then commit the generated files and push to trigger your first CI gate:
Diagnoses local project readiness without cloud access in quick mode. Omit --quick when you also want account and platform connectivity checks.

​
Gate and CI

Compares your current test results against the stored baseline and exits 1 if any metric regresses.
Output format options:
Discovers eval specs, runs them, writes results, and compares against a base run when --base is provided. Add --impacted-only to run only specs affected by the current diff.
Full GitHub Actions workflow:
Runs the regression gate against the EvalGate platform (requires EVALGATE_API_KEY). Use --onFail import to upload failed run context to the dashboard for review.
Re-runs your tests and overwrites the stored baseline with the new results. Run this after you intentionally change model behavior or fix a known issue.

​
Labeling and analysis

Steps through your unlabeled traces one by one. Use arrow keys to select pass/fail and pick a failure mode. Press u to undo the previous label. Press Ctrl-C to save progress and exit.
Each label you save becomes part of the golden dataset that can be used by later eval runs and gates.
Aggregates labeled traces and prints a frequency report of failure modes (counts and share of labeled failures).
Lists standard failure modes used by label, and optionally seeds or extends your evalgate.config.json alert weights.
Custom modes are stored under failureModeAlerts.modes in config and used by analyze and gate alerting.
Loads a candidate eval case from the platform and prints the minimized input for manual or scripted re-run. The legacy replay alias is deprecated — prefer replay-candidate.
Loads two saved run artifacts and emits a keep/discard decision for each case — useful for reviewing whether a prompt change improved or regressed specific failure modes.

​
Agent evidence and enterprise controls

Wraps an agent or CLI subprocess, captures stdout/stderr and tool events, and posts an evidence envelope to EvalGate (unless --local).
Useful flags:
Validates MCP manifests, audits connector readiness, or proxies a server while emitting evidence.
Add --format json for machine-readable audit output. Use --strict-persistence in CI when evidence must land on the platform.
Fetches org-scoped system controls: provider allowlists, PII scrub mode, retention policy, drift alerts, and guardrail state.
Requires EVALGATE_API_KEY with access to /api/system/controls.

​
Repository intelligence and evaluation packs

Use the local activation path first. After the gate is working and GitHub is connected, scan an exact commit, review the evidence, and then install coverage:
Repository scanning never executes repository code. Source findings and pack recommendations remain evidence-bounded and report incomplete scans honestly.

​
Advanced

Reads a saved run artifact and groups cases with similar failure patterns. Use the output to prioritize which failure mode to fix first.
Reads your labeled failure dataset and generates deterministic synthetic test cases to expand coverage of underrepresented failure modes.
Reads labeled failures and prior prompt history, generates the next candidate prompt edit, evaluates it against impacted specs, and keeps the edit only if it does not regress any existing case. The loop terminates on explicit guard conditions rather than running open-ended.
To repeat bounded cycles unattended (for example, overnight):
The TypeScript CLI uses npx @evalgate/sdk auto. The Python CLI exposes the same bounded workflow as evalgate auto run and evalgate auto daemon.
Scans your project for eval spec files, refreshes the manifest, and reports any redundant or overlapping specs.

​
Judge commands

Prints all judges available in the EvalGate registry for your organization.
Prints the built-in judge presets (pre-configured provider + model + prompt combinations).
Runs a judge against a single input/output pair and prints the score, reasoning, and signals. Use this to validate a judge configuration before wiring it into your gate.
Example output:
Runs a judge against two candidate outputs for the same input and returns a preference decision with reasoning. Useful for A/B prompt comparisons.

​
Judge credibility config

Configure judge credibility thresholds and failure-mode alerts in evalgate.config.json at the root of your project:
Set bootstrapSeed to a fixed value (for example, 42) to make judge credibility calculations deterministic across CI runs. Without a fixed seed, bootstrap confidence intervals may vary slightly between runs.
When a judge’s discriminative power (TPR + TNR − 1) falls at or below 0.05, the gate skips score correction and exits with code 8 (WARN) instead of using a potentially biased score. When labeled sample count is below minLabeledSamples, bootstrap confidence intervals are also skipped — both conditions emit reason codes into the judgeCredibility block of the JSON report.

​
Complete command index

Every registered EvalGate CLI command. Run npx @evalgate/sdk <command> --help for the same text in your terminal.