How EvalGate captures regression data
EvalGate does not store one generic “regression” object. It records evidence at three layers, then compares that evidence against a baseline or review decision.Local gate data
The local gate can be created directly withevalgate init --local, including
by a cloud coding agent with no account, browser, organization, GitHub remote,
API key, or network access. Hosted login and link are optional additions for
shared evidence; both paths use the same reviewable local config, specs, and
baseline.
Interactive init previews the scaffold and asks before it writes a pending baseline to evals/baseline.json and installs a GitHub Actions workflow at .github/workflows/evalgate-gate.yml. JSON and non-TTY calls remain preview-only until automation passes explicit --apply; --preview never mutates. Neither local nor hosted-linked init executes tests or accepts the baseline. Run baseline update explicitly after committing the durable scaffold. EvalGate ignores only its managed regenerable run/report artifacts; real dirty source still blocks and is listed by path. The built-in baseline records:
When you run
npx @evalgate/sdk gate, EvalGate runs your test command again, compares the current pass state and test count to the baseline, and writes evals/regression-report.json.
The built-in local gate protects whatever your existing test script measures. If your test script runs AI evals, it gates AI behavior. If it only runs unit tests, it gates test health until you add eval specs or a custom
eval:regression-gate script.Eval run artifacts
The eval runner records behavior at the spec and case level. When you run with--write-results, EvalGate writes:
input, expected, actual, and metadata fields supplied by the spec executor.
New spec run artifacts use schemaVersion: 2. Each intended logical trial has one terminal result. metadata.intendedSpecIds identifies that population; executedSpecs counts attempted trials, while budget-skipped trials remain in the intended population. Retries must be reconciled before aggregation and cannot add another terminal row for the same spec ID.
An assertion-only evaluator may return an unscored observation with nonempty boolean assertion results and a matching overall pass outcome. No numeric score is invented for it. A run with one passing score of 100 and one evaluator exception retains two intended trials, one observation, and one execution failure. Its observed-subset score is 100; its population pass rate is unavailable and it cannot authorize release or become an accepted baseline.
Run indexes retain this scope and completeness. Execution exceptions are excluded from behavioral labels and failure-mode clusters. Legacy version 1 artifacts remain immutable: an explicit execution classification is respected, while an unclassified historical zero has unavailable provenance and cannot establish a behavioral observation.
Run traces and comparison reports also use version 2. Incomplete comparison rows have no winner, and their aggregate scores are labeled as observed subsets. Pending-baseline recommendations require complete passing evidence and no blocking budget outcome. Automatic run-derived harness quality records require version 2 evidence; legacy records are retained without inferring a quality outcome.
Gate reports retain simultaneous evidence, execution, policy, and behavioral blockers. A cost ceiling can block a fully measured passing run without proving behavioral regression.
explain separates observed failures from unconfirmed causes; a score decline or an available prompt alone does not establish prompt drift.
evalgate diff compares a base run and a head run. It classifies changes as new failures, fixed failures, score drops, score improvements, execution errors, added specs, or removed specs.
Platform trace data
The platform captures production and staging behavior through traces and spans.POST /api/collector ingests one trace with spans. POST /api/collector/batch ingests up to 100 traces. The collector stores:
Span identity is trace-scoped:
(trace_id, span_id) identifies one persisted
operation. Exact replays are idempotent, while the same external span_id on a
different trace remains distinct evidence.
After ingestion, EvalGate decides whether to enqueue the trace for failure analysis. Errors and thumbs-down feedback are always analyzed. Successful traces are sampled for analysis at the server default rate of 10%.
Sampling controls failure-analysis work, not whether the SDK sends a trace. The SDK collector helpers send traces by default unless you configure client-side sampling.
From trace to regression coverage
When a trace is analyzed, EvalGate looks for failure signals in the trace output, groups repeated failures by a stable hash, and stores the result as afailure_report. From that report it can create a candidate_eval_case with the source trace IDs, minimized input, expected constraints, quality score, and review status.
Candidate cases start quarantined. They do not gate releases until they are promoted by review or by an explicit promotion workflow. Promoted cases become regression coverage that can be used by eval runs and release gates.
What blocks a release
Different commands block on different evidence:
The most accurate way to describe the loop is: EvalGate captures raw behavior, turns reviewed failures into eval coverage, and blocks releases when current evidence regresses against the selected baseline.