How to Build AI Red-Team Test Cases Before Launch
Turn assets, threat paths, controls, and expected outcomes into reviewed adversarial coverage.
Short answer
Start from protected assets and realistic attacker goals, then map entry points, trust boundaries, possible actions, and expected safe outcomes. Build attack families with variations, preserve provenance, and require review before generated cases become release blockers.
Key takeaways
- Threat models create better cases than generic jailbreak lists.
- Expected safe behavior must be observable and testable.
- Generated attacks remain candidates until reviewed.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- The suite is a copied prompt list with no protected asset or impact.
- Cases score only the final refusal text.
- Attack coverage cannot be traced to a control or threat decision.
Step-by-step approach
- 1
Inventory assets and actions
List sensitive data, identities, tools, money, communications, infrastructure, and decisions the system can affect.
- 2
Map threat paths
Connect attacker goals to input channels, trust boundaries, permissions, and possible side effects.
- 3
Write expected outcomes
Define allow, ask, deny, redact, isolate, or escalate behavior plus independent state checks.
- 4
Generate and review variants
Expand language and delivery methods, deduplicate cases, and have owners approve coverage and severity.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Threat-path coverage | Reviewed asset, entry-point, action, and impact combinations represented by cases. | Prioritize plausible high-impact gaps. |
| Control effectiveness | Attacks blocked or contained by the intended deterministic and model controls. | Investigate passes caused by irrelevant failures. |
| Finding closure proof | Remediated findings with fail-before and pass-after regression evidence. | Do not close on implementation alone. |
Common mistakes
- Equating a larger prompt list with broader risk coverage.
- Generating labels with the same model under test.
- Closing findings without regression proof.
Practical checklist
- Inventory assets and actions: List sensitive data, identities, tools, money, communications, infrastructure, and decisions the system can affect.
- Map threat paths: Connect attacker goals to input channels, trust boundaries, permissions, and possible side effects.
- Write expected outcomes: Define allow, ask, deny, redact, isolate, or escalate behavior plus independent state checks.
- Generate and review variants: Expand language and delivery methods, deduplicate cases, and have owners approve coverage and severity.
Frequently asked questions
What should I do first?
List sensitive data, identities, tools, money, communications, infrastructure, and decisions the system can affect
How should the result be measured?
Reviewed asset, entry-point, action, and impact combinations represented by cases. Prioritize plausible high-impact gaps.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.