Quality Score

Understand how EvalGate computes the 0-100 quality score for every evaluation run, what each component measures, and how to interpret flags and evidence levels.

The quality score is EvalGate’s flagship metric — a single 0-100 number computed for every evaluation run. It captures pass rate, safety, judge quality, schema compliance, latency, and cost into one comparable score so you can track quality trends and gate releases.

How the score is computed

The score is a weighted average of six sub-scores, each normalized to 0-1 before blending:

Component	Weight	What it measures
Pass rate	50%	Fraction of test cases that passed (`passed / total`)
Safety	25%	Safety pass rate from structured assertions or keyword proxy
Judge quality	9% (60% of 15%)	Average LLM judge score, normalized to 0-1
Schema compliance	6% (40% of 15%)	Schema validation pass rate
Latency	6% (60% of 10%)	Average response latency relative to good/bad bands
Cost	4% (40% of 10%)	Average cost per test case relative to budget

Weights are normalized so they always sum to 100%. Organizations can override weights via system policy — see ScoringPolicyConfig.

Formula

score = round(
  (w_passRate * passRate +
   w_safety * safety +
   w_judgeSchema * (split_judge * judge + split_schema * schema) +
   w_latencyCost * (split_latency * latency + split_cost * cost))
  * 100
)

Default values

Parameter	Default	Description
`weights.passRate`	0.50	Weight for pass rate component
`weights.safety`	0.25	Weight for safety component
`weights.judgeSchema`	0.15	Combined weight for judge + schema
`weights.latencyCost`	0.10	Combined weight for latency + cost
`judgeSchemaSplit.judge`	0.60	Judge’s share of the judge+schema block
`judgeSchemaSplit.schema`	0.40	Schema’s share of the judge+schema block
`latencyCostSplit.latency`	0.60	Latency’s share of the latency+cost block
`latencyCostSplit.cost`	0.40	Cost’s share of the latency+cost block
`latencyBands.goodMs`	1500	Latency at or below this scores 1.0
`latencyBands.badMs`	8000	Latency at or above this scores 0.0
`flagThresholds.safetyRisk`	0.95	Safety below this triggers `SAFETY_RISK`
`flagThresholds.lowPassRate`	0.90	Pass rate below this triggers `LOW_PASS_RATE`
`flagThresholds.latencyRisk`	0.50	Latency score below this triggers `LATENCY_RISK`
`flagThresholds.costRisk`	0.50	Cost score below this triggers `COST_RISK`
`evidenceLevel.strongMinN`	10	Minimum test cases for “strong” evidence
`evidenceLevel.mediumMinN`	5	Minimum test cases for “medium” evidence

Latency scoring

Latency is scored linearly between the good and bad bands:

Score 1.0 when avgLatencyMs <= 1500ms
Score 0.0 when avgLatencyMs >= 8000ms
Linear interpolation between the two thresholds

If no latency data is available, the latency sub-score defaults to 1.0 (no penalty).

Cost scoring

Cost is scored relative to the budget:

Score 1.0 when avgCostUsd <= budgetUsd
Linear decay when cost exceeds budget
Score 0.0 when cost reaches 2x budget

If no cost or budget data is available, the cost sub-score defaults to 1.0 (no penalty).

Custom metrics

You can add custom metrics to the score by providing customMetrics in the score inputs. Each custom metric has a value (0-1) and a weight (positive number). Custom metrics are blended with the base score proportionally to their total weight.

Fallback behavior

When a data source is missing, the score uses conservative fallbacks:

Missing data	Fallback
`safetyPassRate`	Falls back to pass rate
`judgeAvg`	Falls back to pass rate
`schemaPassRate`	Falls back to pass rate
`avgLatencyMs`	Defaults to 1.0 (no penalty)
`avgCostUsd` or `budgetUsd`	Defaults to 1.0 (no penalty)

This means a run with only pass/fail data still gets a meaningful score, but the flags will indicate which dimensions are missing.

Flags

Flags are warning indicators surfaced alongside the score. They don’t change the score but alert you to data quality issues or risks:

Flag	Trigger	Meaning
`SAFETY_RISK`	Safety score < 0.95	Safety pass rate is below the acceptable threshold
`LOW_PASS_RATE`	Pass rate < 0.90	Pass rate is below the acceptable threshold
`LATENCY_RISK`	Latency score < 0.50	Average latency is in the poor range
`COST_RISK`	Cost score < 0.50	Average cost is significantly over budget
`MISSING_JUDGE`	No judge data available	LLM judge was not run or results not stored
`MISSING_SAFETY`	No safety data available	Safety assertions were not present
`LOW_N`	Total cases < 10	Sample size is too small for strong evidence
`SAFETY_WEAK_EVIDENCE`	Safety from keyword proxy	Safety was inferred from error text, not structured assertions
`TRACE_COVERAGE_LOW`	Trace coverage < 0.80	Less than 80% of trace-linked cases were matched
`MISSING_PROVENANCE`	No cost records for trace-linked run	Model/provider provenance is missing

Evidence levels

The evidence level indicates how trustworthy the score is based on data completeness:

Level	Requirements	Meaning
Strong	Has judge data, has safety data, and n >= 10	All major dimensions are covered with sufficient sample size
Medium	Has judge or safety data, and n >= 5	Some dimensions covered with moderate sample size
Weak	Missing both judge and safety, or n < 5	Score is based primarily on pass rate with limited evidence

Use the evidence level to decide whether to trust a score for gating decisions. A “weak” score may still be useful for trend tracking but should not block a release on its own.

Scoring spec versioning

Every quality score is persisted with:

scoringVersion — the spec version used (currently v1)
inputsJson — the exact inputs that produced the score
scoringSpecJson — the full scoring spec at computation time
inputsHash — SHA-256 hash of canonicalized inputs
scoringSpecHash — SHA-256 hash of canonicalized scoring spec
scoringCommit — Git commit SHA when the score was computed

This ensures scores are reproducible and auditable. You can verify the scoring formula by fetching /api/quality/spec, which returns the current scoring spec.

API access

GET /api/quality/spec — Returns the current scoring spec (weights, thresholds, version)
GET /api/quality/{evaluationId} — Returns quality scores for an evaluation
POST /api/quality/recompute/{runId} — Recomputes the quality score for a specific run

Get Started

Core Concepts

Guides

SDK Reference

Platform

Quality score

Quality Score

How the score is computed

Formula

Default values

Latency scoring

Cost scoring

Custom metrics

Fallback behavior

Flags

Evidence levels

Scoring spec versioning

API access

​Quality Score

​How the score is computed

​Formula

​Default values

​Latency scoring

​Cost scoring

​Custom metrics

​Fallback behavior

​Flags

​Evidence levels

​Scoring spec versioning

​API access

Quality Score

How the score is computed

Formula

Default values

Latency scoring

Cost scoring

Custom metrics

Fallback behavior

Flags

Evidence levels

Scoring spec versioning

API access