Scorer Studio

Scorer Studio is EvalGate’s organization-scoped registry and evidence workbench for reusable scoring logic. A scorer has a stable key, while every saved definition becomes a new immutable release_artifact version with a numeric versionId, a SHA-256 contentHash, and, for code packages, a signed package hash. Use Scorer Studio when Playground, Experiments, Online Eval, and release gates must execute the same reviewed scoring definition and retain the exact score scale, implementation, prompt/model binding, calibration mapping, and provenance that produced each result.

A consumer binds an exact versionId and contentHash. Scorer Studio never resolves “latest,” silently changes a historical result, or invents a normalized score when this exact scorer version has no reviewer-approved calibration binding with a canonical mappingVersionId.

Permissions and entry points

Open Scorer Studio at /scorers. Select a scorer to open its workbench at /scorers/{scorerId}.

Action	Required scope
List, inspect, compare, read usage and evidence	`eval:read`
Create a scorer/version, add examples, submit for review, bind a consumer	`eval:write`
Execute one-case tests and human-alignment runs	`runs:write`
Approve calibration evidence, approve a version, publish an environment	`scorer:publish`

All reads, writes, composite references, calibration references, results, and surface bindings are organization-scoped. A scorer or referenced version from another organization is rejected rather than disclosed.

Scorer types

Every type returns the same output contract: a finite score, a boolean passed, human-readable reasoning, and a structured evidence object.

Type	Use it for	Versioned definition
Deterministic	Exact/contains/regex matching, JSON Schema checks, latency limits, and cost limits	Operator, configuration, input/output schemas, scale, threshold
Code	Custom TypeScript or Python logic that needs a bounded execution contract	Source, language, `score` entrypoint, CPU/wall-time/memory/output limits, network policy, signed package identity
Composite	Weighted or logical aggregation across reviewed scorers	Aggregation plus each component’s exact artifact ID, version ID, content hash, weight, and required flag
LLM judge	Semantic rubric evaluation	Rubric, judge prompt, JSON response schema, temperature zero, Gateway model/routing configuration, and optional exact Prompt Hub binding

Deterministic regex definitions reject known nested-repetition patterns. Composite versions reject missing components, self-reference, organization mismatch, and content hashes that do not match the referenced immutable version. LLM judge definitions do not accept custom HTTP endpoints. Provider and model egress goes through the authoritative Model Gateway, which applies routing, provider policy, budgets, retries, failure classification, cost attribution, and the model-call ledger.

Author a definition

The Definition tab exposes the complete content-hashed JSON definition. It includes:

Input and output JSON Schemas
Implementation type and its full configuration
Numeric score minimum, maximum, pass threshold, and optional labels
Calibration publication policy and accountable owner
Tags and a concise purpose statement
Exact component or Prompt Hub version bindings where applicable

Choose New scorer, enter a permanent lowercase key, and choose a deterministic, bounded TypeScript, or LLM judge template.
Open the scorer and edit the complete definition.
Resolve every contract issue shown next to the editor.
Choose Create new version.

Saving does not mutate the selected version. It creates a lineage-linked draft with a new numeric ID and content hash. An identical canonical definition resolves idempotently to the existing content version rather than creating an indistinguishable duplicate.

Code execution boundary

Code packages declare allowNetwork: false and explicit limits:

{
  "timeoutMs": 1000,
  "cpuMs": 500,
  "memoryMb": 64,
  "maxOutputBytes": 65536
}

The local development runner rejects module loading, host globals, dynamic code, filesystem access, process creation, and network APIs. It is restricted to non-production trusted development use and requires the explicit SCORER_TRUSTED_LOCAL_CODE_EXECUTION=1 opt-in. Production ignores that setting and never routes to the local executor. Production always routes code scorers through the OCI isolation boundary. Configure SCORER_OCI_RUNTIME as docker or podman and preload the digest-pinned Node and Python images declared by the release. The runner refuses floating image tags, verifies the checked-in seccomp profile hash, and has no production switch back to the local executor. The OCI process runs as UID/GID 65534, with a read-only root filesystem, no network or IPC namespace, no bind mounts, all Linux capabilities dropped, no-new-privileges, a no-exec temporary filesystem, and enforced process, CPU, wall-time, memory/swap, and output ceilings. A default-deny seccomp profile permits runtime threads but denies child-process creation. Only the scorer input, source, limits, and already-verified content/package identity cross stdin; host environment secrets are not copied into the container. Every run records the pinned image digest, seccomp profile hash, content hash, and package hash in its bounded system log. Missing runtimes or images, profile-integrity failure, timeout, output overflow, OOM, invalid output, and scorer exceptions remain distinct failure codes.

Build human anchors

The Examples tab stores append-only human anchors. Each anchor contains:

A name and optional review notes
The full scorer input: input, candidate output, optional expected output, context, latency, and cost
The expected human score on the selected scorer’s scale
Actor and creation time

Editing or deleting a persisted anchor is rejected. Create a new anchor when human judgment changes so historical alignment runs retain their exact input snapshot.

Test one exact version

The Test tab runs one input against the selected versionId. Choose the consuming surface and enter its durable resource ID before running. Every stored result includes:

Organization, scorer artifact, exact version ID, and content hash
Surface and resource ID
Raw score, pass decision, reasoning, structured evidence, and captured logs
Explicit failure code/message, latency, and known or unknown cost
Model-call ID for an LLM judge
Canonical mapping-version ID plus its derived calibration set, run, and legacy mapping IDs when present
Comparability state and normalized score, or an explicit withheld value
Idempotency key and request hash

An LLM judge provider error, budget denial, timeout, or JSON parse failure remains a failed or blocked result with its model-call evidence. It is not presented as a successful score.

Measure human alignment

The Human alignment tab executes a frozen anchor snapshot and returns an evidence matrix with:

Evidence	Meaning
Agreement rate	Pass/fail agreement between scorer and human labels
Correlation	Pearson correlation across successful numeric scores
Confusion counts	True positives, true negatives, false positives, and false negatives
Parse failure rate	Anchors where a judge response could not satisfy its contract
Unstable anchors	Any disagreement or non-successful execution
Latency and cost	Average latency, total known cost, and unknown-cost count
Per-row provenance	Human/scorer values, failure code, model-call ID, latency, and cost

Completed alignment evidence is immutable. Repeating an idempotency key returns the original run instead of creating a conflicting final state.

Calibrate without hiding discontinuities

The Calibration tab separates a raw scorer output from an approved comparison scale. Comparability is one of:

exact_comparable: the exact canonical mapping version on the intended reference scale and an approved binding for this scorer version
calibrated_comparable: a reviewer-approved mapping version on a compatible anchor identity and score scale
trend_discontinuity: the change breaks an earlier score trend
not_comparable: no valid cross-version comparison is claimed

A comparable binding requires an exact canonical mappingVersionId linked to its approved anchor version, judge configuration, calibration run, legacy score mapping, and organization. EvalGate derives and validates the legacy IDs rather than trusting caller-supplied lineage. Supported mappings execute from their immutable stored breakpoints, percentile distribution, or z-score parameters.

Scorer Studio persists normalizedScore = null unless the exact version has an approved exact_comparable or calibrated_comparable mapping. Missing calibration is never treated as zero, one, or “close enough.”

An LLM judge publication policy can require a minimum number of human anchors, minimum agreement, maximum parse-failure rate, an approved scorer calibration binding with a mapping-version ID, and an accountable owner. Publication stops with a specific calibration error when any threshold is unmet.

Compare, review, and approve

The Versions tab lists every immutable version with its status, numeric ID, and content hash. Select a comparison version to inspect a line-level definition diff, including schemas, rubrics, model routing, scale, thresholds, code limits, component hashes, and calibration policy. The lifecycle is:

draft → review → approved

Submitting and approving are compare-and-set writes. The request includes the reviewRevision currently loaded in the workbench. A concurrent transition returns 409 STALE_VERSION; it does not overwrite the newer state. Every transition appends an actor, rationale, revision, timestamp, and idempotency key. Approval does not publish a version. It only makes the exact version eligible for an environment or consumer binding.

Publish and bind consumers

Publishing requires an approved version, the loaded environment version/revision, and an idempotency key. The confirmation dialog names the current and target version IDs and content hash. The environment move and append-only event are atomic; concurrent stale promotions fail without a false event. The Usage tab binds an approved exact version to:

Surface	Resource example
Playground	Playground session or evaluation workspace ID
Experiment	Immutable experiment ID
Online Eval	Monitor/version ID
Gate	Release-gate policy/run ID

Each binding persists surface, resourceId, versionId, contentHash, and the exact canonical mapping version available at creation. Usage results stay attached to the scorer version, mapping version, derived calibration lineage, and model call that produced them after a newer version ships.

API workflow

Method	Endpoint	Purpose
`GET`, `POST`	`/api/scorers`	List or create scorers
`GET`, `PATCH`	`/api/scorers/{scorerId}`	Read workbench data or update metadata
`GET`, `POST`	`/api/scorers/{scorerId}/versions`	List or create immutable versions
`GET`	`/api/scorers/{scorerId}/versions/{versionId}`	Read one exact version
`POST`	`.../{versionId}/submit-review`	Move draft to review
`POST`	`.../{versionId}/approve`	Approve a reviewed version
`POST`	`.../{versionId}/publish`	Atomically bind an environment
`POST`	`.../{versionId}/bindings`	Bind Playground, Experiment, Online Eval, or gate
`POST`	`/api/scorers/{scorerId}/examples`	Append human anchors
`POST`	`/api/scorers/{scorerId}/test`	Execute an exact version
`POST`	`/api/scorers/{scorerId}/alignment-runs`	Run a frozen human-alignment matrix
`GET`	`/api/scorers/{scorerId}/alignment-runs/{runId}`	Read immutable run evidence
`POST`	`/api/scorers/{scorerId}/calibration`	Bind exact calibration/comparability evidence
`GET`	`/api/scorers/{scorerId}/usage`	Read exact surface bindings and results

Version creation, execution, alignment, review, approval, and publishing accept an Idempotency-Key header. Reusing a key for different intent returns a conflict.

Failure behavior

Condition	Result
Invalid schema, score scale, regex, model endpoint, code limit, or component	`400 VALIDATION_ERROR`
Scorer, component, prompt, calibration, or version is outside the organization	`404 NOT_FOUND` or database scope rejection
Review/environment revision changed	`409 STALE_VERSION`
Idempotency key is reused for different input	`409 IDEMPOTENCY_CONFLICT`
Version is not approved for publish or reuse	`409 APPROVAL_REQUIRED`
Human/calibration publication policy is unmet	`409 CALIBRATION_REQUIRED`
Code requests a denied capability	Stored `blocked` result with `SANDBOX_EGRESS_OR_CAPABILITY_BLOCKED`
Production has no configured OCI runtime, pinned image, or intact seccomp profile	Stored `blocked` or failed result with `SANDBOX_PRODUCTION_RUNNER_REQUIRED`, `SANDBOX_RUNTIME_UNAVAILABLE`, `SANDBOX_IMAGE_NOT_PINNED`, `SANDBOX_IMAGE_UNAVAILABLE`, or `SANDBOX_POLICY_INTEGRITY`
Code exceeds a runtime/output bound	Stored `timeout` or `failed` result with the exact limit code
Model Gateway denies budget/policy or provider fails	Stored blocked/failed result with model-call and Gateway error evidence
Judge response does not satisfy the output contract	Stored failed result with `PARSE_FAILURE`

Troubleshooting

If version creation fails, fix the first nearby contract issue and validate the complete definition JSON.
If a composite fails, verify every component artifact/version/hash triplet and required flag.
If an LLM judge is blocked, inspect the model-call ledger for routing, budget, provider, or response-schema evidence.
If publish is blocked, compare the selected version’s human-alignment metrics with its calibration policy and verify that this scorer version’s calibration binding is approved.
If a normalized score is withheld, approve a canonical mapping version and bind its exact mappingVersionId to this scorer version; do not copy legacy calibration IDs or a binding from another version.
If production code execution is blocked, verify SCORER_OCI_RUNTIME, preload the exact Node/Python image digest, keep security/scorer-seccomp.json unchanged, and inspect the stored sandbox error and system log.
If a publication is stale, reload the workbench and review the newer environment revision before retrying.

Verification is maintained in the Scorer Studio unit, hostile-code OCI, API, database, DOM, and authenticated golden-path suites: tests/lib/scorer-studio.test.ts, tests/security/scorer-oci-sandbox.test.ts, tests/api/scorers, tests/integration/scorers, tests/dom/scorers, and e2e/scorer-studio-golden-path.spec.ts. The dedicated Scorer sandbox CI job pulls the same immutable images and runs the non-skipped adversarial boundary suite.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Scorer Studio

Scorer Studio

Permissions and entry points

Scorer types

Author a definition

Code execution boundary

Build human anchors

Test one exact version

Measure human alignment

Calibrate without hiding discontinuities

Compare, review, and approve

Publish and bind consumers

API workflow

Failure behavior

Troubleshooting

​Scorer Studio

​Permissions and entry points

​Scorer types

​Author a definition

​Code execution boundary

​Build human anchors

​Test one exact version

​Measure human alignment

​Calibrate without hiding discontinuities

​Compare, review, and approve

​Publish and bind consumers

​API workflow

​Failure behavior

​Troubleshooting

Scorer Studio

Permissions and entry points

Scorer types

Author a definition

Code execution boundary

Build human anchors

Test one exact version

Measure human alignment

Calibrate without hiding discontinuities

Compare, review, and approve

Publish and bind consumers

API workflow

Failure behavior

Troubleshooting