Skip to main content

​
The trace to eval to gate workflow

EvalGate is built around one operating loop: collect traces from real AI behavior, turn reviewed failures into reusable evaluation cases, and gate regressions in CI before they reach production. Start with the local gate if you only need regression protection. Add the platform when you need shared traces, review history, LLM judges, costs, benchmarks, annotations, and governance. LLMs drift silently. A single prompt change can degrade quality across thousands of responses before anyone notices. The trace to eval to gate loop closes that gap: real failures become reviewed test cases, and promoted test cases become merge gates so the same issue is caught before it ships again.

​
Product layers

​
Trace evidence model

As of June 2026, the strongest AI evaluation stacks treat traces as structured evidence, not just logs. OpenTelemetry’s GenAI semantic conventions have moved into a dedicated GenAI semantic conventions repository, and current evaluation products increasingly connect trace data to datasets, experiments, evaluators, and production monitoring. EvalGate follows that shape by keeping traces useful for three jobs: The goal is portability: keep enough span and metadata structure that traces can feed EvalGate, OpenTelemetry-compatible pipelines, offline datasets, and CI gates without rewriting the evidence each time.
1

Collect traces from real behavior

Traces are structured records of your AI system’s actual behavior in production or staging. Every time your application reports a trace, EvalGate stores the context you send: inputs and outputs, tool calls, token usage, latency, errors, and any metadata you attach.EvalGate uses asymmetric sampling for server-side failure analysis by default: errors and thumbs-down feedback are always analyzed, while successful traces are sampled at 10%. SDK trace reporting sends traces by default unless you configure client-side sampling.
TypeScript
From the Traces page in your dashboard you can search by metadata, inspect nested spans, and identify the exact inputs that caused a failure before converting them into permanent test coverage.
Use descriptive trace names like customer-support-query instead of generic labels like llm-call. Specific names make it faster to find relevant failures when you’re building eval coverage.
2

Turn failures into eval coverage

Raw traces are observations. Evaluations are commitments. Once you have a set of traced failures, you promote them into labeled test cases that run on every change.EvalGate gives you three tools for building eval coverage from traces and run artifacts:Label - Use the interactive CLI to mark each case pass or fail and assign a failure mode. Each saved label becomes part of your golden dataset.
Cluster - Group similar failures by behavior and workflow shape so you can spot patterns instead of triaging individual cases.
Synthesize - Generate additional golden case drafts from labeled failures to expand coverage beyond what production traffic alone produces.
The local synthesize command writes synthetic case drafts to JSONL. Platform-generated candidate eval cases start quarantined and do not gate releases until they are promoted by review or by an explicit promotion workflow.
The result is a reusable evaluation suite that reflects the failure modes your users actually encounter, not hypothetical ones you invented in advance.
3

Gate regressions in CI

With a labeled golden dataset in place, you can enforce quality in CI. The local gate compares your test/eval command against evals/baseline.json. The eval CI path writes run artifacts and can compare a head run against a base run.
A minimal GitHub Actions workflow looks like this:
When a PR introduces a regression, the CI command fails, writes a GitHub step summary, and emits annotations for failed or regressed specs when --format github is used. Reviewers can inspect the run artifacts uploaded from .evalgate/.
Judge credibility is enforced at gate time. If true positive rate (TPR) or true negative rate (TNR) falls below the configured thresholds, the platform gate fails rather than silently using a biased score. The separate weak-discriminative-power case exits with warning code 8. Set tprMin and tnrMin in evalgate.config.json to control the hard thresholds.

​
The full loop

The three steps above map onto a longer operating cycle that runs continuously as your AI system evolves:
The auto stage lets you run bounded prompt-improvement experiments and keep only changes that pass the gate:
Start with trace, gate, and review before using auto. The autonomous loop works best when you already have stable traces, labeled failure modes, and a trusted baseline to gate against.

​
Why this order matters

Running evaluations without traces means testing scenarios you imagined, not the ones users hit. Gating without evaluations means blocking on metrics that don’t reflect real failures. Tracing without gating means collecting signal you never act on. The trace to eval to gate order ensures that every stage feeds the next with real evidence. Issues that reach production can be captured as traces, reviewed into eval coverage, promoted into a gate, and blocked from shipping again.