Skip to main content

​
EvalGate CLI command reference

Complete reference for EvalGate CLI authentication, repository linking, gates, CI integration, trace labeling, failure analysis, judge orchestration, and auto loops.
The EvalGate CLI is the fastest way to run regression gates, analyze failure patterns, and automate prompt improvement without leaving your terminal. Run TypeScript CLI commands with npx @evalgate/sdk <command> for zero-install usage, or add @evalgate/sdk to your project and run the same commands through your package manager.

​
Local core and hosted boundary

The MIT-licensed CLI gives cloud coding agents a credential-free local execution path. init --local performs project discovery and scaffolding with no account, browser, organization, GitHub remote, API key, or network request and creates no hosted identity. Spec execution, baseline updates, and regression gates then run in the checkout. Browser-approved login and durable repository link are a separate optional hosted path. Exact commit and remote-head details remain cloud activation snapshots, not durable link identity. Baselines and local evidence remain reviewable files that can run in any CI provider. The hosted control plane is additive. Connect it for shared history, managed traces, team review, access control, and organization policy. Local gates remain available when the hosted service is not configured.

​
Start a cloud-agent sandbox locally

Here local describes where evaluation executes; the intended caller may be a remote coding agent. JSON and non-TTY init is preview-only without explicit --apply.

​
Add hosted repository collaboration

The normal human path uses browser-approved device authorization and does not print or ask you to copy an EvalGate credential:
link detects the GitHub origin, current symbolic branch, exact local HEAD, and repository root. It previews the server-owned intake contract pinned to that exact target and asks before applying in a TTY. Branch movement does not unlink the project, and EvalGate never silently substitutes the remote branch head. For agents and CI, mutation remains explicit:
Repository configuration cannot supply a key or redirect a user session to another API origin. After installation, deterministic understand, init, health, baseline, and gate commands can run without a network round trip. status keeps durable linkage, checkout movement, cloud snapshot, and readiness separate. A checkout behind its tracked branch can remain linked, locally gate-ready, and cloud-targetable. Ahead or unpushed work needs the exact commit pushed before cloud evidence; diverged work needs an explicit appropriate branch. Dirty worktree state is reported independently because Git SHA alone cannot reproduce uncommitted changes.
The lifecycle on this page follows @evalgate/sdk@3.10.0, including credential-free init --local. PyPI remains at 3.6.0 until its separately governed release closes, so verify the installed Python package before using current TypeScript-parity commands from PyPI.
Both CLIs expose the same versioned capability map. Start here when you know the job you need to accomplish but not the internal command name:
The complete map of governed core workflows covers local product understanding, setup, evaluation, Playground, Prompt Hub, Dataset Hub, first-party evaluation packs, repository intelligence, experiments, scorer development, logs, continuous evaluation, failure topics, dashboards, Copilot, provider onboarding, red-team testing, remote runners, deployable assets, governance, and release. TypeScript and Python emit identical JSON from the generated contract. Every mapped workflow names its web route, public operation IDs, native automation command, artifacts, and next governed step; generation fails when any required workflow disappears.

​
Stable exit codes

The process exit status is the shell/CI result. JSON reports may additionally carry effectiveExitCode and releaseReady when preview mode or evidence policy separates process completion from the release decision. The complete, version-bound contract is available from evalgate capabilities --format json.

​
Call any public API operation

The TypeScript CLI’s api command consumes the generated public operation registry. This gives automation a single command for every operation in the API reference without falling back to private routes or database helpers. Treat it as an escape hatch for advanced automation; evalgate capabilities and the native workflow commands are the primary navigation surface.
Use --path name=value, --query name=value, and --header name=value more than once when an operation declares multiple inputs. --body accepts inline JSON or @path/to/body.json. Machine mode writes the exact successful API envelope to stdout; failures write a stable JSON error containing code, message, status, requestId, and exitCode. Ctrl-C aborts the active request and exits 130.

​
Red-team release evidence

To supply a local release gate with durable Red-Team evidence, request the specific completed run that the release is based on:
The first-class command exists in both SDK CLIs and uses the generated public API operation. It validates the strict artifact and canonical hash, verifies the server’s Ed25519 signature against redTeam.expectedAttestationPublicKeySha256, checks the configured organization, campaign snapshot, and exact release target, then atomically writes .evalgate/red-team-evidence.json. It compiles identity, completion, provider, attack, benign-utility, finding, remediation, golden, signed-report, and framework-reference evidence from that run. It does not execute a campaign or choose the latest run. A required local redTeam policy must pin the exact artifactId, versionId, and 64-character contentHash being released; the gate rejects missing, malformed, stale, mismatched, cross-organization, incomplete, or unverifiable evidence. Do not hand-author the JSON file or substitute a source-checkout digest for the hosted release target. A bare redTeam.required: true uses pre_release_strict: complete and fresh campaign evidence, attributable costs, compiler attestation, zero successful attacks, and 100% benign utility. It intentionally does not require a finding-scoped remediation report, so a clean first campaign can pass. Use post_remediation_strict when admitting a repaired finding; it additionally requires the baseline/fixed rerun comparison, no introduced fingerprints, active golden coverage, and an exact-hash signed report. The legacy strict name remains an alias for pre_release_strict. Pin expectedTarget, expectedOrganizationId, expectedSnapshotHash, expectedAttestationKeyId, and expectedAttestationPublicKeySha256 in evalgate.config.json. Use policyProfile: "custom" only for a reviewed policy exception. A fresh CI job runs evalgate red-team evidence --run <run-id> before evalgate gate; it never receives the server’s private signing key. In machine mode, export failures emit one JSON error envelope. Invalid arguments/configuration exit 5, API/auth/network failures exit 4, artifact integrity or identity-policy failures exit 3, local writes exit 11, and unexpected defects exit 99. Campaign creation, approval, and execution remain explicit actions requiring the appropriate Red-Team permissions. Fetching release evidence is a read-only compilation step.

​
Setup and initialization

Reads repository evidence and immediately returns a working theory of the product, its audience, the user’s job, the primary visible output, likely quality risks, confidence, unknowns, and one useful question. Inferred risks remain hypotheses until a person confirms them or representative behavior provides evidence.Quick mode is the default bounded scan of high-signal product files. Use --depth deep or --deep for a broader local scan. Both modes preview by default.
Human output is the default. --format json returns the complete plan envelope, while --format jsonl emits typed scan, evidence, theory, risk, question, and completion events for agents and automation. --json is an alias for JSON output.Preview writes nothing. It performs no network or model calls, executes no project commands, inspects or changes no Git state, and does not modify baselines, promote cases, or install telemetry. After review, --apply writes the canonical context to evalgate.product.json. A custom --output must remain inside the selected project root.
Product understanding is Experimental in the current source tree. TypeScript and Python CLI parity is locally verified, but package-release evidence, REST and mounted web authoring, marketplace distribution, and controlled coding-agent compatibility proof are still pending.
Detects Node, Python, polyglot, and nested workspace projects. init --local is the credential-free cloud-agent path and creates no hosted context; plain init is the hosted-linked variant. In a TTY bare init previews and asks before applying. Explicit --preview never asks or writes. JSON and non-TTY calls remain non-mutating unless --apply is explicit. Init creates a pending baseline without executing tests or accepting results. It never mints or accepts an implicit activation key.The JSON plan includes project.aiUsage with evidence-backed agent frameworks, model providers, observability libraries, and tool protocols. Declared dependencies are high confidence; exact source imports are medium confidence. init --apply preserves the same evidence under agentRuntimeRegistry.discovery in evalgate.config.json and chooses a default runtime only when one framework is unambiguous.Discovery is local, bounded, and deterministic. It ignores dependency/build/virtual-environment directories, does not read .env files, and never guesses credentials or model names.
Supported first-class handlers are npm, pnpm, the legacy yarn alias, Yarn Classic, Yarn Modern, Bun, Deno, pip, uv, Poetry, Pipenv, PDM, Conda, Mamba, Hatch, Pixi, and explicit custom JSON argv. Conflicting JavaScript or Python/environment markers stop the plan. Repeat --package-handler <id> for an explicit polyglot composition; gate and baseline update execute every runnable handler from its owning root.
baseline update accepts only successful current evidence from the exact clean source that was evaluated. A pending baseline is not release-ready, and generated CI never updates, commits, or pushes one. The local gate writes its canonical release decision to .evalgate/last-report.json and detailed deltas to evals/regression-report.json.Then commit the generated files and push to trigger your first CI gate:
Removed command families fail without mutation using BAD_ARGS (exit 5) and targeted next actions. JSON mode emits one document and redacts secret-like arguments.An API key authorizes automation against an already established hosted context. It cannot bootstrap hosted repository linkage and is unnecessary for the local gate.
Diagnoses local project readiness without cloud access in quick mode. Omit --quick when you also want account and platform connectivity checks.
EvalGate is not limited to AI applications. A small local manifest can check safe HTTP routes, webhooks, background jobs, or command-based smoke tests. Checks run sequentially only when a developer or CI job calls them; no collector, daemon, cloud credential, or long-lived connection is required.
The JSON report contains bounded latency metrics, status/error counts, redacted reasons, and a suggested inspection path. In GitHub Actions, --format github appends the measured reasons to the PR Step Summary. The repository evalgate-application-health Skill helps a coding agent create the manifest and prepare a reviewed change; it does not edit or merge code automatically.Python projects use the parity command:
To evaluate delegation with the versioned evidence extension, see Evaluate your subagents. The existing command accepts an independent expected-root frame:
These comparisons are local and exploratory. Child sessions and retries are not independent root outcomes. Use a build containing the extension; generic exports do not certify native provider context inheritance.EvalGate can improve the harness used to build or operate an application, not only the application’s own AI behavior. Give the CLI a bounded JSONL file containing session outcomes and counters such as input/output tokens, repeated context, redundant tool calls, Skill loads, and cache hits.Each JSONL line contains counters only, for example:
The local report identifies likely waste such as repeated repository discovery, eager Skill loading, redundant reads, or overly long passing summaries. It waits for five outcome-labeled sessions, refuses to optimize when quality is below 70%, and shows a conservative token estimate. It never reads prompts, completions, tool arguments, traces, credentials, or starts a collector. It proposes a change; the existing eval suite must validate the base and candidate before the change is considered validated.
Python exposes the same commands as evalgate harness optimize and evalgate harness feedback. evalgate init --apply installs the portable evalgate-harness-optimization Skill in .agents/skills/ so a coding agent can explain the result and prepare the change without requiring a cloud connector.After the base-vs-candidate eval has passed, record validated feedback, then plan one managed Skill edit:
Apply requires the explicit --apply authority flag and the matching validated report hash. EvalGate changes only its own deterministic managed block, refuses stale/out-of-repository/symlink targets, and writes rollback metadata under .evalgate/harness-mutations/:
Python exposes the same mutation and rollback commands. Mutation remains local and on-demand: no daemon, cloud credentials, raw source snapshot, prompt, completion, or unrelated instruction-file rewrite is involved.

​
Gate and CI

​
Stability policy for probabilistic checks

An LLM call or model-judge verdict is not deterministic just because it runs in CI. Build hard gates on deterministic assertions where possible. For probabilistic checks, pin model and judge configuration, execute independent trials with the SDK’s repetitions option, set an explicit minimumPassRate, and fail or quarantine evidence reported in flakyCases. Retries are transient recovery attempts within a trial; they are surfaced as flaky evidence and should not be used to retry a verdict until it passes. A seed makes EvalGate’s case ordering reproducible but cannot prevent an external provider from changing a model. Recalibrate release-blocking judges against labeled examples, require the CLI’s judge TPR/TNR and minimum-sample thresholds, and keep the resulting credibility evidence with the gate report. Exit code 10 means the configured judge is not credible enough to block a release safely. Before any provider call, the auto editor and SDK model-assertion boundary reject common instruction overrides, role spoofing, and judge-score coercion in untrusted evaluation text. A security rejection fails closed and never falls back to an automatic prompt mutation. The check --onFail import network boundary recursively redacts nested secrets and common PII from the complete failed-run envelope; this convenience upload path has no raw-data opt-out.
Compares your current test results against the stored baseline and exits 1 if any metric regresses.
Output format options:
Discovers eval specs, runs them, writes results, and compares against a base run when --base is provided. Add --impacted-only to run only specs affected by the current diff.
Full GitHub Actions workflow:
Runs the regression gate against the EvalGate platform (requires EVALGATE_API_KEY). Use --onFail import to upload failed run context to the dashboard for review.
Re-runs your tests and overwrites the stored baseline with the new results. Run this after you intentionally change model behavior or fix a known issue.

​
Labeling and analysis

Steps through your unlabeled traces one by one. Use arrow keys to select pass/fail and pick a failure mode. Press u to undo the previous label. Press Ctrl-C to save progress and exit.
Each label you save becomes part of the golden dataset that can be used by later eval runs and gates.
Aggregates labeled traces and prints a frequency report of failure modes (counts and share of labeled failures).
Lists standard failure modes used by label, and optionally seeds or extends your evalgate.config.json alert weights.
Custom modes are stored under failureModeAlerts.modes in config and used by analyze and gate alerting.
Loads a candidate eval case from the platform and prints the minimized input for manual or scripted re-run. The legacy replay alias is deprecated — prefer replay-candidate.
Loads two saved run artifacts and emits a keep/discard decision for each case — useful for reviewing whether a prompt change improved or regressed specific failure modes.

​
Agent evidence and enterprise controls

Wraps an agent or CLI subprocess, captures stdout/stderr and tool events, and posts an evidence envelope to EvalGate (unless --local).
Useful flags:
Validates MCP manifests, audits connector readiness, or proxies a server while emitting evidence.
Add --format json for machine-readable audit output. Critical findings block by default; use --fail-on high (or low, medium, critical) to set the CI threshold. Use --strict-persistence separately when evidence must land on the platform.
Fetches org-scoped system controls: provider allowlists, PII scrub mode, retention policy, drift alerts, and guardrail state.
Requires EVALGATE_API_KEY with access to /api/system/controls.

​
Repository intelligence and evaluation packs

Use the local activation path first. After the gate is working and GitHub is connected, scan an exact commit, review the evidence, and then install coverage:
Repository scanning never executes repository code. Source findings and pack recommendations remain evidence-bounded and report incomplete scans honestly.

​
Advanced

Reads a saved run artifact and groups cases with similar failure patterns. Use the output to prioritize which failure mode to fix first.
Start with contrastive response examples or expand a labeled failure dataset. Profile mode previews locally without writing or calling a model.
Reads labeled failures and prior prompt history, generates the next candidate prompt edit, evaluates it against impacted specs, and keeps the edit only if it does not regress any existing case. The loop terminates on explicit guard conditions rather than running open-ended.
To repeat bounded cycles unattended (for example, overnight):
The TypeScript CLI uses npx @evalgate/sdk auto. The Python CLI exposes the same bounded workflow as evalgate auto run and evalgate auto daemon.
Scans your project for eval spec files, refreshes the manifest, and reports any redundant or overlapping specs.

​
Judge commands

Prints all judges available in the EvalGate registry for your organization.
Prints the built-in judge presets (pre-configured provider + model + prompt combinations).
Runs a judge against a single input/output pair and prints the score, reasoning, and signals. Use this to validate a judge configuration before wiring it into your gate.
Example output:
Runs a judge against two candidate outputs for the same input and returns a preference decision with reasoning. Useful for A/B prompt comparisons.

​
Judge credibility config

Configure judge credibility thresholds and failure-mode alerts in evalgate.config.json at the root of your project:
Set bootstrapSeed to a fixed value (for example, 42) to make judge credibility calculations deterministic across CI runs. Without a fixed seed, bootstrap confidence intervals may vary slightly between runs.
When a judge’s discriminative power (TPR + TNR − 1) falls at or below 0.05, the gate skips score correction and exits with code 8 (WARN) instead of using a potentially biased score. When labeled sample count is below minLabeledSamples, bootstrap confidence intervals are also skipped — both conditions emit reason codes into the judgeCredibility block of the JSON report.

​
Complete command index

Every registered EvalGate CLI command. Run npx @evalgate/sdk <command> --help for the same text in your terminal.