---
title: "How to Catch LLM Regressions in CI"
description: "Run stable fast checks on pull requests and reserve heavier semantic evaluation for the right release stage."
canonical: "https://www.evalgate.com/guides/catch-llm-regressions-in-ci"
date-published: "2026-09-02"
last-updated: "2026-09-02"
category: "Prompt and model changes"
keywords: "LLM regression testing CI, AI evals GitHub Actions, prompt tests CI"
---

# How to Catch LLM Regressions in CI

Run stable fast checks on pull requests and reserve heavier semantic evaluation for the right release stage.

**Question:** How do I catch LLM regressions in continuous integration?

## Short answer

Put deterministic, fast, reproducible checks on every relevant change; run bounded semantic evaluations on representative cases; compare with a reviewed baseline; and emit a nonzero result when protected behavior regresses. Separate lightweight pull-request checks from heavier merge or staging suites.

## Key takeaways

- Map changed AI assets to the smallest relevant suite.
- Keep CI evidence reproducible and attributable to a baseline.
- Fail closed on critical regressions and invalid evidence.

## Signs this is the problem

- Prompt, model, tool, or retrieval changes bypass behavioral tests.
- CI reruns produce different decisions with no explanation.
- The gate posts a score but cannot block a release.

## Step-by-step approach

### 1. Inventory change types

Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior.

### 2. Split the lanes

Run deterministic and small stable suites on pull requests, then broader model-backed suites at merge or staging.

### 3. Pin the evidence

Version cases, scorers, baselines, model settings, and thresholds with the change.

### 4. Emit a real gate

Return pass, warn, or block with failing cases and artifacts visible to reviewers.

## What to measure

| Metric | What it measures | How to use it |
| --- | --- | --- |
| Relevant-change coverage | Share of AI-affecting changes that invoke the correct suite. | Fix bypasses before expanding test volume. |
| Gate reproducibility | Consistency of decisions on unchanged code and evidence. | Quarantine unstable scorers rather than normalizing flaky reruns. |
| Escaped regression rate | Production regressions that the mapped CI suite did not catch. | Add reviewed cases and update change mapping after each escape. |

## Common mistakes

- Running the largest expensive suite on every commit.
- Downloading mutable prompts or thresholds during CI.
- Allowing missing evaluation evidence to appear green.

## Practical checklist

- [ ] Inventory change types: Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior.
- [ ] Split the lanes: Run deterministic and small stable suites on pull requests, then broader model-backed suites at merge or staging.
- [ ] Pin the evidence: Version cases, scorers, baselines, model settings, and thresholds with the change.
- [ ] Emit a real gate: Return pass, warn, or block with failing cases and artifacts visible to reviewers.

## Where EvalGate fits

EvalGate's CLI supports local and CI-formatted regression gates, with documented GitHub Actions generation and a GitLab CI recipe.

[Set up CI/CD regression gates](https://www.evalgate.com/docs/guides/cicd-integration) or [start with the EvalGate quickstart](https://www.evalgate.com/docs/quickstart).

## Frequently asked questions

### What should I do first?

Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior

### How should the result be measured?

Share of AI-affecting changes that invoke the correct suite. Fix bypasses before expanding test volume.

### When is the change ready to ship?

Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.

## Related guides

- [https://www.evalgate.com/guides/test-prompt-changes](https://www.evalgate.com/guides/test-prompt-changes)
- [https://www.evalgate.com/guides/create-ai-release-gate](https://www.evalgate.com/guides/create-ai-release-gate)
- [https://www.evalgate.com/guides/fix-flaky-ai-evals](https://www.evalgate.com/guides/fix-flaky-ai-evals)