Skip to main content

​
Dataset Hub

Dataset Hub turns working evaluation rows into reusable, organization-scoped assets. A dataset can be bound to more than one evaluation without copying mutable source data into historical evidence. The core workflow is:
  1. Create a dataset and define its row schema.
  2. Add rows through the API, CSV, JSONL, JSON, or production-trace conversion.
  3. Resolve validation failures, duplicates, privacy findings, splits, labels, and review state.
  4. Create an immutable version from the active working rows.
  5. Move the version through review, approval, and publication.
  6. Bind the published version to one or more evaluations.
  7. When an evaluation run starts, freeze the exact version, content hash, and row payloads into the run evidence.
Working rows and immutable versions are deliberately different. Editing a working row creates a superseding row with a lineage link. It never rewrites a version or run that already contains the prior payload.

​
Permissions

Viewer controls are disabled in the Dataset Hub UI and a read-only notice explains which actions need elevated access. The API independently enforces scopes and role. Resource lookups always include the authenticated organization; a foreign dataset, row, version, trace, evaluation, or run is never accepted because its ID is syntactically valid.

​
Dataset and row lifecycle

A dataset is either active or archived. Active datasets accept working-row and version changes. Archiving closes row edits, review promotion, imports, and version creation. It does not delete retained versions, bindings, or run snapshots. Working rows carry:
  • input, expected output, context, and metadata;
  • tags, typed labels, and a deterministic split;
  • source type, source reference, and structured source provenance;
  • privacy state and redaction-finding count;
  • review state: draft, candidate, reviewed, or rejected;
  • lifecycle state: active, deprecated, archived, or superseded;
  • a content fingerprint and, for edits, supersedesRowId lineage.
Editing content creates a replacement row and marks the former working row superseded. Review promotion may update only an active row in an active dataset. Dataset versions keep snapshots of both the content and lineage, so version comparison classifies a superseding edit as a changed row instead of an unrelated removal and addition.

​
Row schemas and validation

The dataset schema uses a bounded JSON Schema subset:
  • type: object, array, string, number, integer, boolean, or null;
  • object required, properties, and additionalProperties;
  • array items;
  • enum;
  • string minLength, maxLength, and pattern;
  • numeric minimum and maximum.
Validation runs on the normalized row contract before persistence. A schema error includes the row number, field path, code, and human-readable reason. Changing the working dataset schema affects only later working-row validation and later versions; every existing version retains its original schema snapshot.

​
Imports

POST /api/datasets/{datasetId}/import supports:
  • text/csv with an input header;
  • application/x-ndjson or application/jsonl;
  • a JSON request containing format: csv | jsonl | json and the source content;
  • direct API rows through POST /api/datasets/{datasetId}/rows.
CSV accepts quoted fields, escaped quotes, embedded commas, CRLF, embedded newlines, and JSON-shaped metadata cells. JSONL processes each non-empty line independently. JSON imports must be an array of row objects. Imports are bounded to 10 MB and 50,000 submitted rows. Direct row creation is bounded to 1,000 rows per request. Large jobs should be split into stable chunks and use a distinct idempotency key per chunk. Every import requires an Idempotency-Key. Replaying the same key and content returns the original import report. Reusing the key with different content returns a conflict and does not mutate the dataset.

​
Partial imports

Invalid rows are never silently discarded. An import can finish as:
  • completed: all valid, non-duplicate rows were accepted;
  • partial: at least one row was accepted and at least one failed validation or parsing;
  • failed: no row was accepted;
  • processing: the report exists but final counts have not been recorded yet.
A partial response uses HTTP 207 and includes accepted, duplicate, and failed counts plus row-level failure records. The import panel keeps that report beside the action so an operator can fix rejected rows without guessing which source records were omitted.

​
Deduplication, splits, and privacy

The row fingerprint is a SHA-256 digest of canonical input, expected output, and context. Object-key order and mutable curation metadata do not change the fingerprint. Active duplicate content is inserted once and reported in the duplicate count; concurrent duplicate submissions are also constrained by the database. If a row does not provide a split, Dataset Hub uses its fingerprint to assign a stable default split with 80% train, 10% validation, and 10% test weights. The same row content receives the same split on replay. Explicit split values are preserved. Before any preview or row is persisted, Dataset Hub recursively scans strings in input, expected output, context, metadata, labels, and provenance. Detected email addresses, phone numbers, and supported credential shapes are replaced with [REDACTED]. The stored row exposes clean or redacted state and a finding count; the original sensitive value is not retained in Dataset Hub.
Redaction is a safety boundary, not permission to upload unnecessary personal or secret data. Minimize source payloads before import and rotate any credential that was included in source data.

​
Trace conversion and provenance

POST /api/datasets/{datasetId}/from-traces converts organization-owned trace IDs into candidate rows. The first suitable span input becomes the row input and the last suitable span output becomes the expected output. The row records the public trace ID, database trace reference, source span IDs, environment, trace status, score, and conversion time. If any requested trace belongs to another organization or is missing, the conversion rejects the request rather than importing the subset. Converted rows begin as candidate and still require explicit review. All other row sources record sourceType, optional sourceRef, import time, and structured source provenance. Version summaries count rows by source type.

​
Immutable versions

Version creation takes an organization-and-dataset advisory lock, selects active rows in a deterministic order, and stores:
  • version number;
  • schema snapshot;
  • ordered row snapshots and row hashes;
  • row count and source summary;
  • reviewed and redacted row counts;
  • one SHA-256 content hash for the complete schema and ordered manifest;
  • creator and idempotency key.
Concurrent version creation produces distinct monotonic version numbers. Replaying the same creation key returns the original version. Versions move through a guarded lifecycle:
Skipping a state or returning a published version to a mutable state is rejected. Approval and publication record the actor and time. Transition records are append-only decision evidence. Publishing is serialized and idempotent, so concurrent requests cannot create conflicting publication outcomes. Database triggers reject changes to version schema, row count, content hash, summaries, creation identity, or row snapshots. They also reject update or deletion of version rows and run dataset snapshots.

​
Evaluation bindings and historical runs

Only a published version can be bound to an evaluation. A dataset may be bound to multiple evaluations, while an evaluation has one current binding per dataset. snapshot and tracking modes describe how the current binding is managed; both identify an exact published version at write time. When a run freezes its bindings, Dataset Hub copies the exact version ID, version content hash, and ordered row snapshot into evaluation_run_dataset_snapshots. Repeating the freeze is idempotent. Later working-row edits, a v2 publication, or a current-binding change cannot reinterpret the v1 payload executed by the historical run. Use GET /api/evaluations/{evaluationId}/runs/{runId}/dataset-snapshots to inspect the frozen evidence.

​
Version comparison

Choose a base and comparison version in the UI or call:
The response separates added, removed, and changed row IDs and reports whether the schema changed. Lineage traversal follows multiple generations of supersedesRowId, so a row edited between v1 and v3 remains a change to its original lineage even if v2 contained an intermediate replacement.

​
Dataset health

The health endpoint and UI summarize actionable quality signals rather than a pass/fail gate:
  • exact and near duplicates;
  • empty, extreme-length, missing-output, score, and schema outliers;
  • schema drift;
  • label and split distributions;
  • reviewed and redacted row counts;
  • a health score, summary, and prioritized recommendations.
Health can guide curation, but publication remains an explicit reviewed decision. A high health score does not make candidate or generated rows trusted evidence.

​
API reference

List endpoints use opaque cursors ordered by creation time and ID. Invalid cursors return validation errors. A response cursor never embeds organization authority; the authenticated organization remains the query boundary.

​
Failure behavior

​
Troubleshooting

​
Import is partial

Open the import report and correct each line by its failure code and field path. Keep the original report for evidence, then submit a new source payload with a new idempotency key. Reusing the original key with changed content is intentionally rejected.

​
A row is reported as a duplicate

Compare input, expected output, and context. Tags, labels, source metadata, and review state do not create a new content identity. If the content truly changed, edit the existing working row to create a lineage-preserving replacement.

​
Publish is unavailable

Confirm that the version is in approved, you have admin or owner access, and the dataset is active. Progress through review and approval rather than skipping lifecycle states.

​
An old run does not show v2 rows

This is expected. Historical runs resolve their frozen run snapshot, not the current dataset binding. Inspect the run’s dataset snapshot and content hash to reproduce the executed v1 payload.

​
Health score changed after curation

Health analyzes active working rows. It can change when rows are superseded, reviewed, redacted, relabeled, or re-split. Published version summaries and historical run snapshots remain unchanged.

​
Verification evidence

  • Contract and parser tests cover schema validation, canonical fingerprints, deterministic splits, content hashes, cursors, redaction, malformed files, size limits, and partial JSONL.
  • API tests cover malformed requests, bounded imports, partial reports, idempotency, foreign trace IDs, redacted output, and cursor pagination.
  • Database tests cover immutable snapshots, row lineage, concurrent versioning and publication, append-only decisions, multi-evaluation bindings, historical run reproducibility, organization isolation, idempotent imports, redaction, archive, and retention.
  • UI tests cover empty, permission, import-partial, review, health, privacy, provenance, and version-diff states.