Create a scorer
A scorer turns a case’s output into a score. EvalGate supports four kinds. Start
simple and add complexity only when you need it.
Pick the right kind
Step 1 — Open Scorer Studio
In the sidebar, click Scorers, then New scorer.
Step 2 — Define the scorer
Pick the kind, then fill in the definition:
- Deterministic — choose a comparator and the expected value.
- Code — paste a TypeScript or Python function. Production runs inside the
OCI sandbox; local runs are for development only.
- Composite — add component scorers and weights. Each component references an
exact version.
- LLM judge — write the judge prompt, pick a model, and set the score
mapping.
Step 3 — Add examples
On the Examples tab, add labeled input/output pairs. These are the
ground-truth cases the scorer must agree with.
Step 4 — Test
On the Test tab, run the scorer against the examples. Every example must
match its label before you submit for review.
Step 5 — Submit and publish
Submit for review. A reviewer with scorer:publish approves the version, which
becomes immutable and content-hashed. Bind it to an evaluation to use it.
Code scorers run in a restricted, digest-pinned OCI sandbox in production. Do
not assume your local Node or Python environment matches production exactly.
Up next
Calibrate the scorer so its scores are
comparable across versions.