> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Scorer Studio
> Author, test, align, calibrate, approve, publish, and reuse immutable scorer versions.
# Scorer Studio
Scorer Studio is EvalGate's organization-scoped registry and evidence workbench for reusable scoring logic. A scorer has a stable key, while every saved definition becomes a new immutable `release_artifact` version with a numeric `versionId`, a SHA-256 `contentHash`, and, for code packages, a signed package hash.
Use Scorer Studio when Playground, Experiments, Online Eval, and release gates must execute the same reviewed scoring definition and retain the exact score scale, implementation, prompt/model binding, calibration mapping, and provenance that produced each result.
A consumer binds an exact `versionId` and `contentHash`. Scorer Studio never resolves “latest,” silently changes a historical result, or invents a normalized score when this exact scorer version has no reviewer-approved calibration binding with a canonical `mappingVersionId`.
## Permissions and entry points
Open **Scorer Studio** at `/scorers`. Select a scorer to open its workbench at `/scorers/{scorerId}`.
| Action | Required scope |
| ------------------------------------------------------------------------- | ---------------- |
| List, inspect, compare, read usage and evidence | `eval:read` |
| Create a scorer/version, add examples, submit for review, bind a consumer | `eval:write` |
| Execute one-case tests and human-alignment runs | `runs:write` |
| Approve calibration evidence, approve a version, publish an environment | `scorer:publish` |
All reads, writes, composite references, calibration references, results, and surface bindings are organization-scoped. A scorer or referenced version from another organization is rejected rather than disclosed.
## Scorer types
Every type returns the same output contract: a finite `score`, a boolean `passed`, human-readable `reasoning`, and a structured `evidence` object.
| Type | Use it for | Versioned definition |
| ------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- |
| Deterministic | Exact/contains/regex matching, JSON Schema checks, latency limits, and cost limits | Operator, configuration, input/output schemas, scale, threshold |
| Code | Custom TypeScript or Python logic that needs a bounded execution contract | Source, language, `score` entrypoint, CPU/wall-time/memory/output limits, network policy, signed package identity |
| Composite | Weighted or logical aggregation across reviewed scorers | Aggregation plus each component's exact artifact ID, version ID, content hash, weight, and required flag |
| LLM judge | Semantic rubric evaluation | Rubric, judge prompt, JSON response schema, temperature zero, Gateway model/routing configuration, and optional exact Prompt Hub binding |
Deterministic regex definitions reject known nested-repetition patterns. Composite versions reject missing components, self-reference, organization mismatch, and content hashes that do not match the referenced immutable version.
LLM judge definitions do not accept custom HTTP endpoints. Provider and model egress goes through the authoritative Model Gateway, which applies routing, provider policy, budgets, retries, failure classification, cost attribution, and the model-call ledger.
## Author a definition
The **Definition** tab exposes the complete content-hashed JSON definition. It includes:
* Input and output JSON Schemas
* Implementation type and its full configuration
* Numeric score minimum, maximum, pass threshold, and optional labels
* Calibration publication policy and accountable owner
* Tags and a concise purpose statement
* Exact component or Prompt Hub version bindings where applicable
1. Choose **New scorer**, enter a permanent lowercase key, and choose a deterministic, bounded TypeScript, or LLM judge template.
2. Open the scorer and edit the complete definition.
3. Resolve every contract issue shown next to the editor.
4. Choose **Create new version**.
Saving does not mutate the selected version. It creates a lineage-linked draft with a new numeric ID and content hash. An identical canonical definition resolves idempotently to the existing content version rather than creating an indistinguishable duplicate.
### Code execution boundary
Code packages declare `allowNetwork: false` and explicit limits:
```json theme={null}
{
"timeoutMs": 1000,
"cpuMs": 500,
"memoryMb": 64,
"maxOutputBytes": 65536
}
```
The local development runner rejects module loading, host globals, dynamic code, filesystem access, process creation, and network APIs. It is restricted to non-production trusted development use and requires the explicit `SCORER_TRUSTED_LOCAL_CODE_EXECUTION=1` opt-in. Production ignores that setting and never routes to the local executor.
Production always routes code scorers through the OCI isolation boundary. Configure `SCORER_OCI_RUNTIME` as `docker` or `podman` and preload the digest-pinned Node and Python images declared by the release. The runner refuses floating image tags, verifies the checked-in seccomp profile hash, and has no production switch back to the local executor.
The OCI process runs as UID/GID `65534`, with a read-only root filesystem, no network or IPC namespace, no bind mounts, all Linux capabilities dropped, `no-new-privileges`, a no-exec temporary filesystem, and enforced process, CPU, wall-time, memory/swap, and output ceilings. A default-deny seccomp profile permits runtime threads but denies child-process creation. Only the scorer input, source, limits, and already-verified content/package identity cross stdin; host environment secrets are not copied into the container.
Every run records the pinned image digest, seccomp profile hash, content hash, and package hash in its bounded system log. Missing runtimes or images, profile-integrity failure, timeout, output overflow, OOM, invalid output, and scorer exceptions remain distinct failure codes.
## Build human anchors
The **Examples** tab stores append-only human anchors. Each anchor contains:
* A name and optional review notes
* The full scorer input: input, candidate output, optional expected output, context, latency, and cost
* The expected human score on the selected scorer's scale
* Actor and creation time
Editing or deleting a persisted anchor is rejected. Create a new anchor when human judgment changes so historical alignment runs retain their exact input snapshot.
## Test one exact version
The **Test** tab runs one input against the selected `versionId`. Choose the consuming surface and enter its durable resource ID before running.
Every stored result includes:
* Organization, scorer artifact, exact version ID, and content hash
* Surface and resource ID
* Raw score, pass decision, reasoning, structured evidence, and captured logs
* Explicit failure code/message, latency, and known or unknown cost
* Model-call ID for an LLM judge
* Canonical mapping-version ID plus its derived calibration set, run, and legacy mapping IDs when present
* Comparability state and normalized score, or an explicit withheld value
* Idempotency key and request hash
An LLM judge provider error, budget denial, timeout, or JSON parse failure remains a failed or blocked result with its model-call evidence. It is not presented as a successful score.
## Measure human alignment
The **Human alignment** tab executes a frozen anchor snapshot and returns an evidence matrix with:
| Evidence | Meaning |
| ------------------ | -------------------------------------------------------------------- |
| Agreement rate | Pass/fail agreement between scorer and human labels |
| Correlation | Pearson correlation across successful numeric scores |
| Confusion counts | True positives, true negatives, false positives, and false negatives |
| Parse failure rate | Anchors where a judge response could not satisfy its contract |
| Unstable anchors | Any disagreement or non-successful execution |
| Latency and cost | Average latency, total known cost, and unknown-cost count |
| Per-row provenance | Human/scorer values, failure code, model-call ID, latency, and cost |
Completed alignment evidence is immutable. Repeating an idempotency key returns the original run instead of creating a conflicting final state.
## Calibrate without hiding discontinuities
The **Calibration** tab separates a raw scorer output from an approved comparison scale.
Comparability is one of:
* `exact_comparable`: the exact canonical mapping version on the intended reference scale and an approved binding for this scorer version
* `calibrated_comparable`: a reviewer-approved mapping version on a compatible anchor identity and score scale
* `trend_discontinuity`: the change breaks an earlier score trend
* `not_comparable`: no valid cross-version comparison is claimed
A comparable binding requires an exact canonical `mappingVersionId` linked to its approved anchor version, judge configuration, calibration run, legacy score mapping, and organization. EvalGate derives and validates the legacy IDs rather than trusting caller-supplied lineage. Supported mappings execute from their immutable stored breakpoints, percentile distribution, or z-score parameters.
Scorer Studio persists `normalizedScore = null` unless the exact version has an approved `exact_comparable` or `calibrated_comparable` mapping. Missing calibration is never treated as zero, one, or “close enough.”
An LLM judge publication policy can require a minimum number of human anchors, minimum agreement, maximum parse-failure rate, an approved scorer calibration binding with a mapping-version ID, and an accountable owner. Publication stops with a specific calibration error when any threshold is unmet.
## Compare, review, and approve
The **Versions** tab lists every immutable version with its status, numeric ID, and content hash. Select a comparison version to inspect a line-level definition diff, including schemas, rubrics, model routing, scale, thresholds, code limits, component hashes, and calibration policy.
The lifecycle is:
```text theme={null}
draft → review → approved
```
Submitting and approving are compare-and-set writes. The request includes the `reviewRevision` currently loaded in the workbench. A concurrent transition returns `409 STALE_VERSION`; it does not overwrite the newer state. Every transition appends an actor, rationale, revision, timestamp, and idempotency key.
Approval does not publish a version. It only makes the exact version eligible for an environment or consumer binding.
## Publish and bind consumers
Publishing requires an approved version, the loaded environment version/revision, and an idempotency key. The confirmation dialog names the current and target version IDs and content hash. The environment move and append-only event are atomic; concurrent stale promotions fail without a false event.
The **Usage** tab binds an approved exact version to:
| Surface | Resource example |
| ----------- | --------------------------------------------- |
| Playground | Playground session or evaluation workspace ID |
| Experiment | Immutable experiment ID |
| Online Eval | Monitor/version ID |
| Gate | Release-gate policy/run ID |
Each binding persists `surface`, `resourceId`, `versionId`, `contentHash`, and the exact canonical mapping version available at creation. Usage results stay attached to the scorer version, mapping version, derived calibration lineage, and model call that produced them after a newer version ships.
## API workflow
| Method | Endpoint | Purpose |
| -------------- | ------------------------------------------------ | ------------------------------------------------- |
| `GET`, `POST` | `/api/scorers` | List or create scorers |
| `GET`, `PATCH` | `/api/scorers/{scorerId}` | Read workbench data or update metadata |
| `GET`, `POST` | `/api/scorers/{scorerId}/versions` | List or create immutable versions |
| `GET` | `/api/scorers/{scorerId}/versions/{versionId}` | Read one exact version |
| `POST` | `.../{versionId}/submit-review` | Move draft to review |
| `POST` | `.../{versionId}/approve` | Approve a reviewed version |
| `POST` | `.../{versionId}/publish` | Atomically bind an environment |
| `POST` | `.../{versionId}/bindings` | Bind Playground, Experiment, Online Eval, or gate |
| `POST` | `/api/scorers/{scorerId}/examples` | Append human anchors |
| `POST` | `/api/scorers/{scorerId}/test` | Execute an exact version |
| `POST` | `/api/scorers/{scorerId}/alignment-runs` | Run a frozen human-alignment matrix |
| `GET` | `/api/scorers/{scorerId}/alignment-runs/{runId}` | Read immutable run evidence |
| `POST` | `/api/scorers/{scorerId}/calibration` | Bind exact calibration/comparability evidence |
| `GET` | `/api/scorers/{scorerId}/usage` | Read exact surface bindings and results |
Version creation, execution, alignment, review, approval, and publishing accept an `Idempotency-Key` header. Reusing a key for different intent returns a conflict.
## Failure behavior
| Condition | Result |
| --------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Invalid schema, score scale, regex, model endpoint, code limit, or component | `400 VALIDATION_ERROR` |
| Scorer, component, prompt, calibration, or version is outside the organization | `404 NOT_FOUND` or database scope rejection |
| Review/environment revision changed | `409 STALE_VERSION` |
| Idempotency key is reused for different input | `409 IDEMPOTENCY_CONFLICT` |
| Version is not approved for publish or reuse | `409 APPROVAL_REQUIRED` |
| Human/calibration publication policy is unmet | `409 CALIBRATION_REQUIRED` |
| Code requests a denied capability | Stored `blocked` result with `SANDBOX_EGRESS_OR_CAPABILITY_BLOCKED` |
| Production has no configured OCI runtime, pinned image, or intact seccomp profile | Stored `blocked` or failed result with `SANDBOX_PRODUCTION_RUNNER_REQUIRED`, `SANDBOX_RUNTIME_UNAVAILABLE`, `SANDBOX_IMAGE_NOT_PINNED`, `SANDBOX_IMAGE_UNAVAILABLE`, or `SANDBOX_POLICY_INTEGRITY` |
| Code exceeds a runtime/output bound | Stored `timeout` or `failed` result with the exact limit code |
| Model Gateway denies budget/policy or provider fails | Stored blocked/failed result with model-call and Gateway error evidence |
| Judge response does not satisfy the output contract | Stored failed result with `PARSE_FAILURE` |
## Troubleshooting
* If version creation fails, fix the first nearby contract issue and validate the complete definition JSON.
* If a composite fails, verify every component artifact/version/hash triplet and required flag.
* If an LLM judge is blocked, inspect the model-call ledger for routing, budget, provider, or response-schema evidence.
* If publish is blocked, compare the selected version's human-alignment metrics with its calibration policy and verify that this scorer version's calibration binding is approved.
* If a normalized score is withheld, approve a canonical mapping version and bind its exact `mappingVersionId` to this scorer version; do not copy legacy calibration IDs or a binding from another version.
* If production code execution is blocked, verify `SCORER_OCI_RUNTIME`, preload the exact Node/Python image digest, keep `security/scorer-seccomp.json` unchanged, and inspect the stored sandbox error and system log.
* If a publication is stale, reload the workbench and review the newer environment revision before retrying.
Verification is maintained in the Scorer Studio unit, hostile-code OCI, API, database, DOM, and authenticated golden-path suites: `tests/lib/scorer-studio.test.ts`, `tests/security/scorer-oci-sandbox.test.ts`, `tests/api/scorers`, `tests/integration/scorers`, `tests/dom/scorers`, and `e2e/scorer-studio-golden-path.spec.ts`. The dedicated Scorer sandbox CI job pulls the same immutable images and runs the non-skipped adversarial boundary suite.