How to Test an AI Application for PII Leakage
Probe memorization, retrieval, tool results, logs, cross-tenant access, and output redaction without putting real people at risk.
Short answer
Use synthetic canary identities and authorized test records to probe every data path: prompts, retrieval, tools, memory, logs, caches, exports, and outputs. Verify tenant and scope boundaries deterministically, then test whether the model reveals, transforms, or infers protected data.
Key takeaways
- Use controlled canaries instead of exposing real personal data.
- Test storage and telemetry as well as model output.
- Authorization must prevent access before redaction tries to clean it.
Signs this is the problem
Start by confirming the symptom before changing prompts, models, or infrastructure.
- Tests inspect chat output but not traces, logs, or exports.
- One tenant's identifier can be supplied while authenticated to another.
- Sensitive tool results enter context even when the final answer masks them.
Step-by-step approach
- 1
Map sensitive data flows
Document where personal data enters, moves, persists, appears in telemetry, and leaves the system.
- 2
Seed safe canaries
Create synthetic records with unique markers across tenants, scopes, roles, and retention classes.
- 3
Probe access and disclosure
Test direct requests, inference, indirect injection, tool misuse, cross-tenant IDs, logs, caches, and exports.
- 4
Verify deletion and retention
Confirm expiration, redaction, deletion, and evidence handling across every stored representation.
What to measure
| Metric | What it measures | How to use it |
|---|---|---|
| Unauthorized disclosure rate | Attempts that expose any protected canary outside its allowed context. | Treat one verified cross-tenant leak as critical. |
| Pre-output exposure | Sensitive values entering prompts, traces, or tools unnecessarily. | Fix collection and access before relying on output filters. |
| Retention compliance | Representations removed or transformed according to the declared policy. | Verify actual storage rather than configuration labels. |
Common mistakes
- Using live customer data in adversarial prompts.
- Assuming output redaction fixes upstream overcollection.
- Testing only the happy-path application UI.
Practical checklist
- Map sensitive data flows: Document where personal data enters, moves, persists, appears in telemetry, and leaves the system.
- Seed safe canaries: Create synthetic records with unique markers across tenants, scopes, roles, and retention classes.
- Probe access and disclosure: Test direct requests, inference, indirect injection, tool misuse, cross-tenant IDs, logs, caches, and exports.
- Verify deletion and retention: Confirm expiration, redaction, deletion, and evidence handling across every stored representation.
Frequently asked questions
What should I do first?
Document where personal data enters, moves, persists, appears in telemetry, and leaves the system
How should the result be measured?
Attempts that expose any protected canary outside its allowed context. Treat one verified cross-tenant leak as critical.
When is the change ready to ship?
Ship only after the protected cases pass, the primary metric clears its agreed boundary, and the team reviews the remaining failure modes instead of relying on one aggregate score.