Skip to main content

​
Prompt Hub

Prompt Hub is EvalGate’s organization-scoped registry for reusable message prompts. A prompt has a stable key, while every saved definition becomes a new immutable version with its own numeric versionId, monotonically increasing version number, and SHA-256 contentHash. Use Prompt Hub when the same prompt must remain exactly attributable across Playground tests, experiments, traces, deployments, evaluations, online evaluations, model calls, and Evals-as-Code.
An environment alias resolves one exact versionId and contentHash. Production and staging never fall back to a draft or to the newest version.

​
Permissions and entry points

Open Prompt Hub at /prompts. Select a prompt to open its version workbench at /prompts/{promptId}. All reads and writes are organization-scoped. A prompt or version from another organization is not disclosed.

​
What a prompt version contains

The workbench treats the following fields as one content-hashed definition:
  • Ordered messages with stable IDs, roles, optional names, and template content
  • Typed variables with required flags, descriptions, and JSON defaults
  • JSON Schema input and optional output contracts
  • Provider, model, or routing-key defaults resolved by the Model Gateway
  • Generation parameters such as temperature, token limit, and seed
  • Tool names, descriptions, strictness, and JSON Schema input contracts
  • Response format, data policy, fallback policy, and tags
Saving any change creates a new version linked to its lineage parent. It does not update the selected version in place. Database constraints and triggers also reject mutation or deletion of persisted immutable prompt payloads and their identity records.

​
Author a complete definition

  1. Open /prompts and choose New prompt.
  2. Enter a stable lowercase key such as support.answer, a display name, and the initial message template.
  3. Open the prompt and use Messages to add, remove, or edit ordered message roles, IDs, names, tool-call IDs, and content.
  4. Use Configure to author variables, input/output schemas, tools, model defaults, parameters, response format, tags, and runtime policies.
  5. Resolve every {{variable}} reference and correct every schema/configuration warning.
  6. Choose Save as new version. The workbench records the selected version as the lineage parent.
Variable edits synchronize the corresponding input-schema property and required list. You can also edit the full JSON Schema directly. Invalid JSON, a non-object input schema, duplicate or invalid variable/tool names, mismatched required fields, undeclared template variables, and a missing output schema for json_schema block the save.

​
Secrets and custom endpoints

Do not place provider keys, bearer tokens, or other literal secrets in message or tool content. Prompt validation rejects common literal-secret patterns. Store provider credentials in Model Gateway configuration and refer to durable configuration keys. Provider, model, and routing fields do not accept URL values. Custom inference endpoints must be configured through the Model Gateway, which applies its egress and SSRF validation before a prompt can use them.

​
Test an exact version

The Test tab accepts a JSON object that must satisfy the selected version’s input schema. A test:
  1. Resolves defaults and fails if a required value remains unresolved.
  2. Renders the selected immutable message list.
  3. Sends the rendered messages, model/routing defaults, parameters, tools, and response format through the Model Gateway.
  4. Writes a model-call ledger record and a playground prompt-usage link.
  5. Returns the exact prompt versionId, contentHash, resolved messages, model-call ID, status, latency, and cost attribution.
Cost is shown as Unknown when the Gateway cannot attribute it. Unknown is never displayed or aggregated as zero. A provider failure remains a failed model call with durable provenance; it is not reported as a successful prompt test.

​
Compare, review, and approve

Use Compare to compare a version with its lineage parent. The semantic diff reports changed definition paths and message additions, removals, or edits. Approval warnings call out required-variable additions, type changes, variable/tool removals, model/routing changes, and output-schema changes. The lifecycle is:
Submitting and approving are compare-and-set writes. Send the reviewRevision that you read. If another actor transitions the version first, the stale request fails with 409 instead of overwriting the newer state. Each transition appends a review-history event with actor, rationale, revisions, timestamp, and idempotency key. Approval does not publish a version. It only makes the exact version eligible for a protected environment.

​
Publish to an environment

Use Deploy to inspect environment aliases. Publishing requires:
  • The exact target versionId
  • The alias expectedBindingVersionId currently loaded by the client, or null when unbound
  • The loaded expectedBindingRevision
  • A unique idempotency key
staging, prod, and production are protected. They accept approved versions only. The alias move and append-only environment event are written atomically. A stale expected version or revision returns 409; no partial history event is written. The Usage tab makes the current production and staging bindings visible with exact version IDs, full content hashes, and alias revisions. An unbound protected environment explicitly reports that no version resolves.

​
Roll back safely

Rollback changes only the environment alias; it never edits or deletes either prompt version.
  1. Open Deploy and choose Rollback for a bound environment.
  2. Confirm the exact current and target version IDs in the dialog.
  3. Submit the loaded binding version and revision.
The service atomically points the alias to its recorded previous version, increments the revision, and appends a rollback event. Concurrent stale rollback attempts fail without moving the alias or adding a false event. Historical usage stays attached to the version that produced it.

​
Resolve and render in an application

Resolve by prompt artifact and environment:
The response includes the stable key, environment, review state, complete definition, exact version ID, version number, and content hash. If the environment is unbound, the request returns 404 ENVIRONMENT_UNBOUND; it does not return a draft. Render with typed input:
The render response preserves the same versionId and contentHash and adds resolvedMessages. Invalid or unresolved input returns a validation error before model egress.

​
Cross-surface attribution

Every consumer records both versionId and contentHash; a key or environment name alone is not sufficient evidence. Usage insertion rejects a content hash that does not match the referenced immutable version. Repeating the same surface/resource/version identity is idempotent. The Usage view aggregates quality, attributed cost, unknown-cost samples, and latency per exact version rather than merging different versions under the stable key.

​
API workflow

Mutations that create versions or history accept an Idempotency-Key header. Reusing a key for a different payload returns an idempotency conflict.

​
Failure behavior

​
Troubleshooting

  • If save is disabled, inspect the validation panel in Messages and Configure. Fix invalid JSON before changing tabs.
  • If render fails, compare the submitted object with the exact version’s input schema and required variables.
  • If approval or publish is stale, reload the workbench and use the latest review or alias revision. Do not blindly retry with a changed expectation.
  • If a test fails, open its model-call ledger link and inspect Gateway routing, provider health, budget/policy decision, and redacted provider error.
  • If cost is unknown, confirm the selected model exists in the synchronized catalog with current pricing metadata.
  • If production is unbound, approve a version and publish it explicitly. Creating or saving a newer draft will not change production.
Verification is maintained in the Prompt Hub unit, API, database, DOM, and authenticated golden-path suites: tests/unit/prompts, tests/api/prompts, tests/integration/prompts, tests/dom/prompts, and e2e/prompt-hub-golden-path.spec.ts.