> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Failure Topics > Discover recurring production failures, inspect their evidence, govern topic definitions, and turn approved findings into active regression coverage. # Failure Topics Failure Topics turns authoritative failure reports into operational cohorts. It groups recurring failure modes, preserves the evidence and classifier lineage behind every membership, and connects an accepted topic to Dataset Hub, Playground, Review, candidate eval cases, monitors, and active regression coverage. **Status:** Beta · **Owner:** Production Intelligence · **Last verified:** 3.7.0-rc (2026-07-13) ## Open the workspace Open `/insights`. The workspace has two panes: * The topic queue shows status, severity, recurring/novel/severe cohorts, occurrence count, affected users, known cost, confidence, and trend state. * Topic detail shows impact and deployment segments, membership evidence, definition versions, Model Gateway provenance, human decisions, and downstream corrective resources. Readers need trace-read permission. Discovery, acceptance, edits, and corrective actions require trace-write permission and at least the member role. Merge and split require an admin with organization-admin permission. Suppression is a durable audited decision; it does not delete evidence. ## Discover topics from failure evidence Select **Discover from failure reports** to analyze open, unsuppressed reports from the previous 30 days. The request is bounded to 1,000 report rows in the UI and 2,000 at the API contract. Narrow the lookback window if the query exceeds that bound. Discovery requires at least three occurrences by default. A failure report can represent multiple deduplicated occurrences, so its persisted `occurrenceCount` contributes to the evidence threshold without inventing additional membership rows. If no semantic cluster meets the threshold, discovery returns `INSUFFICIENT_EVIDENCE` and leaves the existing workspace unchanged. The deterministic classifier: 1. orders reports by stable case identity before clustering; 2. normalizes failure descriptions into the canonical failure taxonomy; 3. assigns reports with deterministic embeddings and a configured similarity threshold; 4. fingerprints the normalized failure mode rather than its changing member list; and 5. persists new memberships idempotently. Rediscovery does not duplicate a topic or rewrite prior membership history. Topic fingerprints remain stable when report order changes or new occurrences reuse the same normalized failure mode. ## Interpret cohorts and impact Topics can belong to more than one cohort: | Cohort | Meaning | | ----------- | ---------------------------------------------------------------------------------------- | | `recurring` | The topic represents at least three occurrences. | | `novel` | Its first evidence appeared inside the configured novelty window, seven days by default. | | `severe` | Its highest supported severity is high or critical. | Impact snapshots distinguish known and unknown data. Report and trace counts, unique affected users/workflows, model, prompt, and deployment segments come from owned trace and failure-report provenance. Cost is summed only from priced Model Gateway ledger records. If none are priced, the UI shows **Unknown**, not `$0`. Trend values compare persisted snapshots. A merge or split starts a new continuity window and displays **Trend reset**; EvalGate does not present pre-change and post-change membership as a continuous time series. Low-volume history displays **Not enough history** instead of a percentage. ## Inspect membership evidence Open **Evidence** to inspect each active membership. A row includes: * the owned trace key or failure-report ID; * the canonical category and detector sources; * assignment source and confidence; * deduplicated occurrence count; * a redacted evidence hash; and * source-membership IDs after a merge or split. The workspace never sends raw prompts or responses to the label-proposal workflow. Memberships store a hash and explicit redaction state. An organization cannot attach its topic to another organization's report, trace, alert, topic version, action, or membership; database invariants fail closed on those references. ## Review and version a topic Deterministic discovery creates a proposed topic definition. Review its membership evidence before selecting **Accept proposal**. Acceptance records the actor, expected version, rationale, and before/after state, then activates the exact proposed version. Human rename, description, severity, status, and owner changes create a new accepted append-only version. They do not edit a historical version. Every decision uses optimistic concurrency. If another operator changes the topic first, the request returns `STALE_PROPOSAL`; refresh and review the new version before deciding. ### Request an AI label proposal Select **Propose clearer label**, then choose a configured Model Gateway provider and model. EvalGate sends only redacted categories, severity, detector names, confidence, and evidence hashes through the canonical gateway. The resulting name, description, and severity are stored with the gateway `modelCallId` as a **proposal**. AI-generated labels never activate themselves. A human must inspect the current evidence and accept the proposal. Invalid structured output returns `GATEWAY_OUTPUT_INVALID` and does not create a version. ## Merge, split, suppress, and resolve ### Merge overlapping topics Select two or more topics, open the intended target, and merge. The request includes the expected version of every source and the target. EvalGate creates a new accepted target version, copies memberships with source-membership lineage, marks old memberships inactive, records decisions on every participating topic, and marks sources as merged. The target starts a new trend continuity window. ### Split a mixed topic In **Evidence**, select the memberships for the first partition and choose **Split selected**. Every active membership must appear exactly once across at least two non-empty partitions. Unknown, duplicated, or omitted membership IDs return `INVALID_SPLIT` without changing the topic. Each child receives a stable fingerprint, an accepted split definition, copied membership lineage, and a discontinuous first snapshot. The source remains visible with `split` status for audit history. ### Suppress or resolve Suppression requires a reason and confirmation. It removes the topic from active operational work while retaining versions, evidence, snapshots, decisions, and downstream links. Resolve a topic when the failure mode is addressed; reopen it by moving it back to active with a new human version and rationale. ## Create corrective coverage Corrective actions use owned active trace memberships and are idempotent. A repeated idempotency key returns the stored result; reusing the key with different input returns `IDEMPOTENCY_CONFLICT`. | Action | Result | | ---------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Dataset | Creates a Dataset Hub dataset and adds evidence-backed trace rows. | | Playground | Creates or reuses trace-derived test cases in the selected evaluation and opens its Playground. | | Review | If candidate links exist, creates governed `synthetic_candidate` review items; otherwise creates trace review items. | | Candidates | Creates candidate eval cases in the selected evaluation with trace and failure provenance. | | Promote approved | Promotes only candidates with an approved adjudication decision into active-gate test cases. Unapproved candidates fail locally with `approved_review_required`. | | Monitor | Creates a draft Continuous Eval monitor from the topic's canonical failure category. Activate it only after testing its match and budget policy. | Bulk actions preserve successful items when another item fails. The receipt becomes `partial`, item-level failures remain visible, and completed resource links are retained. A partial response is not reported as complete success. The governed regression path is: 1. create candidate eval cases from a topic; 2. create candidate review items; 3. record an approved human adjudication decision; 4. promote approved candidates; and 5. verify the linked active-gate test case in topic history. Promotion cannot bypass Review. The topic action checks an approved adjudication source, and the canonical synthetic promotion service independently enforces the candidate lifecycle transition and provenance contract. ## API workflow | Purpose | Operation | | ---------------------- | ----------------------------------------------------------------------------------------------- | | Summary | `GET /api/production-insights/summary` | | List topics | `GET /api/production-insights/topics` with `status`, `severity`, `cohort`, `query`, and `limit` | | Discover | `POST /api/production-insights/discover` | | Topic evidence | `GET /api/production-insights/topics/{topicId}` | | Human edit | `PATCH /api/production-insights/topics/{topicId}` | | Accept proposal | `POST /api/production-insights/topics/{topicId}/accept` | | Gateway label proposal | `POST /api/production-insights/topics/{topicId}/propose-label` | | Merge | `POST /api/production-insights/topics/merge` | | Split | `POST /api/production-insights/topics/{topicId}/split` | | Corrective action | `POST /api/production-insights/topics/{topicId}/actions/{action}` | The complete workflow is currently available through authenticated REST and the web workspace. The generated TypeScript and Python SDKs and CLI do not yet expose dedicated Failure Topics clients. ## Failure and recovery | Symptom | Meaning | Recovery | | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- | | `INSUFFICIENT_EVIDENCE` | Fewer than the required occurrences formed a qualifying cluster, or no owned trace supports an action. | Ingest/analyze more production failures or lower the threshold only when your operating policy allows it. | | `TOO_MANY_REPORTS` | Discovery exceeded its bounded source-row limit. | Narrow the lookback window; do not cluster a silent partial sample. | | `STALE_PROPOSAL` | A topic version changed after the operator loaded it. | Refresh, inspect the current evidence/version, and resubmit the decision. | | `INVALID_MERGE` or `INVALID_SPLIT` | The requested lineage operation is incomplete, duplicated, or version-stale. | Correct the selected topics or membership partitions and retry with current versions. | | Action returns `partial` | Some trace, candidate, or destination items failed while others completed. | Inspect item errors and retry only failed inputs with a new idempotency key. | | Promotion says `approved_review_required` | No approved candidate adjudication exists. | Open Review, record an approved decision, then promote again. | | Trend shows **Trend reset** | A merge or split changed the classifier membership boundary. | Treat the new window as a fresh series; inspect pre-change snapshots in History. | | Cost shows **Unknown** | No priced Model Gateway ledger record supports the selected evidence. | Repair cost provenance; do not substitute zero. | ## Verification evidence The release-candidate acceptance evidence covers: * [domain and clustering tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/unit/failure-topics.test.ts) for order stability, fingerprints, cohorts, impact, novelty, continuity, and split validation; * [authenticated route tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/api/failure-topics.routes.test.ts) for validation, scope/role enforcement, optimistic versions, and action contracts; * [PostgreSQL integration proof](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/integration/failure-topics/failure-topics.db.test.ts) for deterministic rediscovery, idempotent membership, and tenant isolation; * [workspace DOM tests](https://github.com/evalgate/ai-evaluation-platform/blob/main/tests/dom/failure-topics/failure-topics-workspace.test.tsx) for evidence, provenance, empty/insufficient states, and trend discontinuity; and * [authenticated golden path](https://github.com/evalgate/ai-evaluation-platform/blob/main/e2e/failure-topics-golden-path.spec.ts) from recurring production failure through accepted topic, candidate, governed review, promotion gate, and active regression coverage.