Skip to main content

​
Your first local gate

This is the fastest proof that EvalGate is useful. You will create a baseline, add a CI step, then watch the gate fail when you break something on purpose. No account, no API key, no dashboard.

​
What you need

  • A repo with at least one test or eval command that prints pass/fail.
  • A terminal, and permission to push to GitHub.

​
Step 1 — Plan the setup

In your repo, run:
This prints what EvalGate would create without writing anything. Read the plan. It should mention a baseline file, a config file, and a CI workflow.

​
Step 2 — Apply the setup

When the plan looks right, apply it:
You now have the committed setup artifacts:

​
Step 3 — Accept a baseline

The applied baseline is pending — it has not run your suite yet. Run your suite once and accept the result as the baseline:

​
Step 4 — Push and open a PR

Open a pull request. The CI workflow runs your suite, compares it to the baseline, and passes.

​
Step 5 — Break something on purpose

Change a test or eval so it fails, then push:
The same CI workflow now fails with a regression report that names what changed. That is the whole loop: baseline → change → regression report → CI gate.

​
Troubleshooting

  • The gate passed when it should have failed. You probably did not run baseline update after init --apply. The baseline is still pending.
  • The gate failed on the first run. Your suite was already failing when you accepted the baseline. Fix the suite, then re-run baseline update.
  • Nothing happened in CI. Confirm .github/workflows/evalgate-gate.yml was committed and pushed to the default branch.

​
Up next

When you are ready to feed real production failures into the same loop, connect a model provider key.