How to Catch LLM Regressions in CI
Run stable fast checks on pull requests and reserve heavier semantic evaluation for the right release stage.
Short answer
Put deterministic, fast, reproducible checks on every relevant change; run bounded semantic evaluations on representative cases; compare with a reviewed baseline; and emit a nonzero result when protected behavior regresses. Separate lightweight pull-request checks from heavier merge or staging suites.
Key takeaways
- Map changed AI assets to the smallest relevant suite.
- Keep CI evidence reproducible and attributable to a baseline.
- Fail closed on critical regressions and invalid evidence.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Prompt, model, tool, or retrieval changes bypass behavioral tests.
- CI reruns produce different decisions with no explanation.
- The gate posts a score but cannot block a release.
Step-by-step approach
- 1
Inventory change types
Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior.
- 2
Split the lanes
Run deterministic and small stable suites on pull requests, then broader model-backed suites at merge or staging.
- 3
Pin the evidence
Version cases, scorers, baselines, model settings, and thresholds with the change.
- 4
Emit a real gate
Return pass, warn, or block with failing cases and artifacts visible to reviewers.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Relevant-change coverage | Share of AI-affecting changes that invoke the correct suite. | Fix bypasses before expanding test volume. |
| Gate reproducibility | Consistency of decisions on unchanged code and evidence. | Quarantine unstable scorers rather than normalizing flaky reruns. |
| Escaped regression rate | Production regressions that the mapped CI suite did not catch. | Add reviewed cases and update change mapping after each escape. |
Common mistakes
- Running the largest expensive suite on every commit.
- Downloading mutable prompts or thresholds during CI.
- Allowing missing evaluation evidence to appear green.
Practical checklist
- Inventory change types: Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior.
- Split the lanes: Run deterministic and small stable suites on pull requests, then broader model-backed suites at merge or staging.
- Pin the evidence: Version cases, scorers, baselines, model settings, and thresholds with the change.
- Emit a real gate: Return pass, warn, or block with failing cases and artifacts visible to reviewers.
Frequently asked questions
What should I do first?
Map prompt, model, tool schema, retrieval, policy, and orchestration changes to affected behavior
How should the result be measured?
Share of AI-affecting changes that invoke the correct suite. Fix bypasses before expanding test volume.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.