​

Failure Topics

Failure Topics turns authoritative failure reports into operational cohorts. It groups recurring failure modes, preserves the evidence and classifier lineage behind every membership, and connects an accepted topic to Dataset Hub, Playground, Review, candidate eval cases, monitors, and active regression coverage. Status: Beta · Owner: Production Intelligence · Last verified: 3.7.0-rc (2026-07-13)

​

Open the workspace

Open /insights. The workspace has two panes:

• The topic queue shows status, severity, recurring/novel/severe cohorts, occurrence count, affected users, known cost, confidence, and trend state.
• Topic detail shows impact and deployment segments, membership evidence, definition versions, Model Gateway provenance, human decisions, and downstream corrective resources.

Readers need trace-read permission. Discovery, acceptance, edits, and corrective actions require trace-write permission and at least the member role. Merge and split require an admin with organization-admin permission. Suppression is a durable audited decision; it does not delete evidence.

​

Discover topics from failure evidence

Select Discover from failure reports to analyze open, unsuppressed reports from the previous 30 days. The request is bounded to 1,000 report rows in the UI and 2,000 at the API contract. Narrow the lookback window if the query exceeds that bound. Discovery requires at least three occurrences by default. A failure report can represent multiple deduplicated occurrences, so its persisted occurrenceCount contributes to the evidence threshold without inventing additional membership rows. If no semantic cluster meets the threshold, discovery returns INSUFFICIENT_EVIDENCE and leaves the existing workspace unchanged. The deterministic classifier:

1. orders reports by stable case identity before clustering;
2. normalizes failure descriptions into the canonical failure taxonomy;
3. assigns reports with deterministic embeddings and a configured similarity threshold;
4. fingerprints the normalized failure mode rather than its changing member list; and
5. persists new memberships idempotently.

Rediscovery does not duplicate a topic or rewrite prior membership history. Topic fingerprints remain stable when report order changes or new occurrences reuse the same normalized failure mode.

​

Interpret cohorts and impact

Topics can belong to more than one cohort: Impact snapshots distinguish known and unknown data. Report and trace counts, unique affected users/workflows, model, prompt, and deployment segments come from owned trace and failure-report provenance. Cost is summed only from priced Model Gateway ledger records. If none are priced, the UI shows Unknown, not $0. Trend values compare persisted snapshots. A merge or split starts a new continuity window and displays Trend reset; EvalGate does not present pre-change and post-change membership as a continuous time series. Low-volume history displays Not enough history instead of a percentage.

​

Inspect membership evidence

Open Evidence to inspect each active membership. A row includes:

• the owned trace key or failure-report ID;
• the canonical category and detector sources;
• assignment source and confidence;
• deduplicated occurrence count;
• a redacted evidence hash; and
• source-membership IDs after a merge or split.

The workspace never sends raw prompts or responses to the label-proposal workflow. Memberships store a hash and explicit redaction state. An organization cannot attach its topic to another organization’s report, trace, alert, topic version, action, or membership; database invariants fail closed on those references.

​

Review and version a topic

Deterministic discovery creates a proposed topic definition. Review its membership evidence before selecting Accept proposal. Acceptance records the actor, expected version, rationale, and before/after state, then activates the exact proposed version. Human rename, description, severity, status, and owner changes create a new accepted append-only version. They do not edit a historical version. Every decision uses optimistic concurrency. If another operator changes the topic first, the request returns STALE_PROPOSAL; refresh and review the new version before deciding.

​

Request an AI label proposal

Select Propose clearer label, then choose a configured Model Gateway provider and model. EvalGate sends only redacted categories, severity, detector names, confidence, and evidence hashes through the canonical gateway. The resulting name, description, and severity are stored with the gateway modelCallId as a proposal. AI-generated labels never activate themselves. A human must inspect the current evidence and accept the proposal. Invalid structured output returns GATEWAY_OUTPUT_INVALID and does not create a version.

​

Merge, split, suppress, and resolve

​

Merge overlapping topics

Select two or more topics, open the intended target, and merge. The request includes the expected version of every source and the target. EvalGate creates a new accepted target version, copies memberships with source-membership lineage, marks old memberships inactive, records decisions on every participating topic, and marks sources as merged. The target starts a new trend continuity window.

​

Split a mixed topic

In Evidence, select the memberships for the first partition and choose Split selected. Every active membership must appear exactly once across at least two non-empty partitions. Unknown, duplicated, or omitted membership IDs return INVALID_SPLIT without changing the topic. Each child receives a stable fingerprint, an accepted split definition, copied membership lineage, and a discontinuous first snapshot. The source remains visible with split status for audit history.

​

Suppress or resolve

Suppression requires a reason and confirmation. It removes the topic from active operational work while retaining versions, evidence, snapshots, decisions, and downstream links. Resolve a topic when the failure mode is addressed; reopen it by moving it back to active with a new human version and rationale.

​

Create corrective coverage

Corrective actions use owned active trace memberships and are idempotent. A repeated idempotency key returns the stored result; reusing the key with different input returns IDEMPOTENCY_CONFLICT. Bulk actions preserve successful items when another item fails. The receipt becomes partial, item-level failures remain visible, and completed resource links are retained. A partial response is not reported as complete success. The governed regression path is:

1. create candidate eval cases from a topic;
2. create candidate review items;
3. record an approved human adjudication decision;
4. promote approved candidates; and
5. verify the linked active-gate test case in topic history.

Promotion cannot bypass Review. The topic action checks an approved adjudication source, and the canonical synthetic promotion service independently enforces the candidate lifecycle transition and provenance contract.

​

API workflow

The complete workflow is currently available through authenticated REST and the web workspace. The generated TypeScript and Python SDKs and CLI do not yet expose dedicated Failure Topics clients.

​

Failure and recovery

​

Verification evidence

The release-candidate acceptance evidence covers:

• domain and clustering tests for order stability, fingerprints, cohorts, impact, novelty, continuity, and split validation;
• authenticated route tests for validation, scope/role enforcement, optimistic versions, and action contracts;
• PostgreSQL integration proof for deterministic rediscovery, idempotent membership, and tenant isolation;
• workspace DOM tests for evidence, provenance, empty/insufficient states, and trend discontinuity; and
• authenticated golden path from recurring production failure through accepted topic, candidate, governed review, promotion gate, and active regression coverage.