Set up CI/CD regression gates with EvalGate

EvalGate has three CI paths. Choose the one that matches the evidence you want to gate on.

Path	Account required	Command	Blocks on
Local gate	No	`npx @evalgate/sdk gate`	Existing test/eval script compared with `evals/baseline.json`
Eval artifact CI	No	`npx @evalgate/sdk ci`	Eval spec failures and optional base/head run diff
Platform gate	Yes	`npx @evalgate/sdk check`	Platform score, baseline drop, policies, budgets, and judge credibility

One-command eval artifact CI

Use ci when your repo contains EvalGate spec files and you want each pull request to write .evalgate/ run artifacts.

.github/workflows/evalgate.yml

name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx @evalgate/sdk ci --format github --write-results --base main
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-results
          path: .evalgate/

That command:

Discovers evaluation specs automatically
Runs all specs by default
Writes run artifacts when --write-results is set
Compares against the base branch when --base is set
Writes a GitHub step summary and annotations when --format github is set
Exits with the appropriate code (see exit codes)

Add --impacted-only to run only specs affected by the current diff. Impact analysis requires a base branch, so use it with --base main or another base ref.

Local gate setup

Initialize local evaluation without a hosted prerequisite. A cloud coding agent needs no browser, account, organization, GitHub remote, or EvalGate API key. Add login and link separately only when the workflow needs shared hosted evidence; every cloud intake is then pinned to its exact target commit.

Human TTY

npx @evalgate/sdk init --local
npx @evalgate/sdk baseline update

In an interactive terminal, init previews and asks before writing the reviewed files. Confirm the prompt to apply; then baseline update runs your configured suite and accepts only a non-empty passing baseline.

Agent or automation

npx @evalgate/sdk init --local --format json
# review the exact plan
npx @evalgate/sdk init --local --apply --format json
npx @evalgate/sdk baseline update

JSON and other non-interactive invocations remain preview-only until explicit --apply. Explicit init --preview never prompts, writes, or runs project commands, even in a TTY. init --local performs no network request and creates no hosted identity. The deterministic local gate remains credential-free. If hosted linkage is later added, a checkout behind its tracked branch is still linked and can run the local gate while cloud evidence remains pinned to the exact requested SHA. Commit the generated files:

git add evalgate.project.json evalgate.config.json evals/ .github/workflows/evalgate-gate.yml
git commit -m "chore: add EvalGate regression gate"
git push

The generated workflow uses the local gate command:

.github/workflows/evalgate-gate.yml

name: EvalGate Gate
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
jobs:
  regression-gate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
          cache: npm
      - run: npm ci
      - name: EvalGate regression gate
        run: npx -y @evalgate/sdk gate --format github --offline
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-report
          path: .evalgate/last-report.json
          if-no-files-found: error

The canonical .evalgate/last-report.json is the release decision envelope; evals/regression-report.json is the detailed regression artifact. Baselines are reviewed evidence and are never automatically updated, committed, or pushed by generated CI. Accept a baseline explicitly after reviewing a clean run with evalgate baseline update.

The built-in local gate compares test pass state and parsed test count against the baseline. If your test script runs AI evals, this gates AI behavior. Otherwise, it gates test health until you add eval specs or a custom eval:regression-gate script.

Platform gate

Use check when you want the dashboard, quality scores, judge credibility, and import-on-failure behavior.

.github/workflows/evalgate-platform-gate.yml

name: EvalGate Platform Gate
on:
  pull_request:
    branches: [main]
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
      - run: npm ci
      - name: EvalGate quality gate
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}
        run: npx @evalgate/sdk check --format github --onFail import

check --format github emits GitHub annotations for failed cases and writes a step summary with the verdict, score, baseline score, delta, and top failing cases. --onFail import uploads the failed run context to the platform so you can review it in the dashboard.

GitLab CI configuration

For GitLab, use JSON output and upload the report as an artifact:

.gitlab-ci.yml

eval-gate:
  stage: test
  image: node:20
  script:
    - npm ci
    - npx @evalgate/sdk gate --format json
  artifacts:
    when: always
    paths:
      - evals/regression-report.json
  only:
    - merge_requests
    - main

Setting quality thresholds

Define local gate thresholds in evalgate.config.json. The built-in local gate uses evals/baseline.json; platform gates use the quality and judge settings configured for the evaluation.

evalgate.config.json

{
  "gate": {
    "baseline": "evals/baseline.json",
    "report": "evals/regression-report.json"
  },
  "judge": {
    "tprMin": 0.7,
    "tnrMin": 0.7,
    "minLabeledSamples": 30
  }
}

Store evalgate.config.json and evals/baseline.json in version control. Changing thresholds or baselines without review changes what your CI gate proves.

Exit codes

The process exit status is the shell/CI result. JSON reports may additionally carry effectiveExitCode and releaseReady when preview mode or evidence policy separates process completion from the release decision. The complete, version-bound contract is available from evalgate capabilities --format json.

Code	Symbol	Meaning	Release meaning
`0`	`PASS`	All required evidence passed the configured gate.	Release-ready when the report also says `releaseReady=true`.
`1`	`SCORE_BELOW`	The measured score is below the configured minimum.	Blocks release.
`2`	`REGRESSION`	Measured behavior regressed beyond the accepted baseline tolerance.	Blocks release.
`3`	`POLICY_VIOLATION`	A configured policy check failed.	Blocks release.
`4`	`API_ERROR`	The hosted API or network request failed.	Release readiness is inconclusive; retry or use an admissible cache.
`5`	`BAD_ARGS`	The command arguments are invalid.	Blocks release until configuration is corrected.
`6`	`LOW_N`	The measured sample is too small for the configured decision.	Blocks release unless policy explicitly allows weak evidence.
`7`	`WEAK_EVIDENCE`	Evidence exists but is not strong enough for the gate.	Blocks release.
`8`	`WARN_REGRESSION`	A metric entered the configured warning band.	Policy-dependent; never treat as a stronger pass than reported.
`9`	`FAILURE_MODE_THRESHOLD`	A failure-mode threshold was exceeded.	Blocks release.
`10`	`JUDGE_CREDIBILITY_UNTRUSTWORTHY`	Judge calibration is too weak to support a trusted decision.	Blocks release or remains inconclusive per policy.
`11`	`IO_ERROR`	Required report or evidence could not be persisted.	Blocks release; a missing canonical report is not a pass.
`12`	`BASELINE_TAMPER`	The baseline checksum or attestation is invalid.	Blocks release until the reviewed baseline is restored.
`13`	`PROVIDER_UNAVAILABLE`	Required provider evidence was unavailable or stale.	Inconclusive or blocked; never release-green.
`14`	`PROVIDER_INCONCLUSIVE`	Provider work was deferred under an explicit policy.	Inconclusive unless policy explicitly admits deferred evidence.
`15`	`BASELINE_NOT_ACCEPTED`	The current run is measurable, but no reviewed baseline has been accepted.	Blocks release until a reviewed baseline is accepted.
`99`	`INTERNAL_ERROR`	An unexpected internal error occurred.	Blocks release.
`130`	`CANCELLED`	The operation was cancelled by the user or orchestrator.	No release decision was made.

These meanings are authoritative for the installed runtime. Agents and CI integrations can read the complete version-bound contract with npx @evalgate/sdk capabilities --format json.

CLI commands reference

npx @evalgate/sdk init --local     # interactive preview, confirmation, and write
npx @evalgate/sdk baseline update
npx @evalgate/sdk doctor --quick

npx @evalgate/sdk init --local --format json
# review the exact plan
npx @evalgate/sdk init --local --apply --format json
npx @evalgate/sdk baseline update
npx @evalgate/sdk gate --format json

npx evalgate login
npx evalgate link
npx evalgate status --json

npx @evalgate/sdk gate
npx @evalgate/sdk gate --format github
npx @evalgate/sdk gate --format json

npx @evalgate/sdk ci --format github --write-results --base main
npx @evalgate/sdk ci --format github --write-results --base main --impacted-only

npx @evalgate/sdk check --format github --onFail import
npx @evalgate/sdk check --format json

Best practices

Start local

Use the local gate first so every repo gets a baseline before the team adopts platform workflows.

Upload artifacts

Keep .evalgate/ or evals/regression-report.json as CI artifacts so failures are debuggable after the job exits.

Use impacted-only deliberately

Add --impacted-only only after your spec manifest is stable. Unknown changes fall back to broader runs.

Promote reviewed cases

Treat generated candidate cases as drafts until a human or promotion workflow approves them for gating.

Get Started

Core Concepts

Guides

SDK Reference

Platform

CI/CD regression gates

Set up CI/CD regression gates with EvalGate

One-command eval artifact CI

Local gate setup

Human TTY

Agent or automation

Platform gate

GitLab CI configuration

Setting quality thresholds

Exit codes

CLI commands reference

Best practices

Start local

Upload artifacts

Use impacted-only deliberately

Promote reviewed cases

​Set up CI/CD regression gates with EvalGate

​One-command eval artifact CI

​Local gate setup

​Human TTY

​Agent or automation

​Platform gate

​GitLab CI configuration

​Setting quality thresholds

​Exit codes

​CLI commands reference

​Best practices

Start local

Upload artifacts

Use impacted-only deliberately

Promote reviewed cases

Set up CI/CD regression gates with EvalGate

One-command eval artifact CI

Local gate setup

Human TTY

Agent or automation

Platform gate

GitLab CI configuration

Setting quality thresholds

Exit codes

CLI commands reference

Best practices