# How to Turn Production Agent Failures into Regression Tests

Production failures become useful regression coverage only when the failure is
attributable, reviewed, and reproducible.

## Start with one attributable interaction

Preserve the input, relevant retrieval or tool steps, observable output,
expected behavior, and the first point where execution departed from it. Redact
secrets and customer identifiers before treating production evidence as test
data.

## Convert the symptom into reusable behavior

A production trace contains incidental details. Protect the underlying behavior
instead: for example, require a verified eligibility signal before an agent can
call a refund tool. Keep the source identity and reviewer decision as
provenance.

## Use a trace-to-gate workflow

1. Capture the smallest trace that proves the failure.
2. Explain whether the cause was retrieval, model behavior, tool selection,
   arguments, policy, or infrastructure.
3. Author a reviewed case with explicit pass criteria.
4. Prove the case detects the known-bad behavior.
5. Run the affected scope against an immutable baseline.

Do not automatically promote raw production data into a release blocker.
Provider timeouts, parser failures, and missing candidates are execution
failures, not product-quality evidence.

## Report the whole release tradeoff

A useful gate report covers quality, protected slices, reliability, latency,
and cost. Use an explicit `not_measured` state rather than inventing a metric.

Start with one repeated failure you can reproduce. See the
[trace-to-eval guide](https://www.evalgate.com/docs/concepts/trace-eval-gate)
and [CI guide](https://www.evalgate.com/docs/guides/cicd-integration).