> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Quality score

# Quality Score

> Understand how EvalGate computes the 0-100 quality score for every evaluation run, what each component measures, and how to interpret flags and evidence levels.

The quality score is EvalGate's flagship metric — a single 0-100 number computed for every evaluation run. It captures pass rate, safety, judge quality, schema compliance, latency, and cost into one comparable score so you can track quality trends and gate releases.

## How the score is computed

The score is a weighted average of six sub-scores, each normalized to 0-1 before blending:

| Component             | Weight          | What it measures                                             |
| --------------------- | --------------- | ------------------------------------------------------------ |
| **Pass rate**         | 50%             | Fraction of test cases that passed (`passed / total`)        |
| **Safety**            | 25%             | Safety pass rate from structured assertions or keyword proxy |
| **Judge quality**     | 9% (60% of 15%) | Average LLM judge score, normalized to 0-1                   |
| **Schema compliance** | 6% (40% of 15%) | Schema validation pass rate                                  |
| **Latency**           | 6% (60% of 10%) | Average response latency relative to good/bad bands          |
| **Cost**              | 4% (40% of 10%) | Average cost per test case relative to budget                |

Weights are normalized so they always sum to 100%. Organizations can override weights via system policy — see `ScoringPolicyConfig`.

### Formula

```
score = round(
  (w_passRate * passRate +
   w_safety * safety +
   w_judgeSchema * (split_judge * judge + split_schema * schema) +
   w_latencyCost * (split_latency * latency + split_cost * cost))
  * 100
)
```

### Default values

| Parameter                    | Default | Description                                      |
| ---------------------------- | ------- | ------------------------------------------------ |
| `weights.passRate`           | 0.50    | Weight for pass rate component                   |
| `weights.safety`             | 0.25    | Weight for safety component                      |
| `weights.judgeSchema`        | 0.15    | Combined weight for judge + schema               |
| `weights.latencyCost`        | 0.10    | Combined weight for latency + cost               |
| `judgeSchemaSplit.judge`     | 0.60    | Judge's share of the judge+schema block          |
| `judgeSchemaSplit.schema`    | 0.40    | Schema's share of the judge+schema block         |
| `latencyCostSplit.latency`   | 0.60    | Latency's share of the latency+cost block        |
| `latencyCostSplit.cost`      | 0.40    | Cost's share of the latency+cost block           |
| `latencyBands.goodMs`        | 1500    | Latency at or below this scores 1.0              |
| `latencyBands.badMs`         | 8000    | Latency at or above this scores 0.0              |
| `flagThresholds.safetyRisk`  | 0.95    | Safety below this triggers `SAFETY_RISK`         |
| `flagThresholds.lowPassRate` | 0.90    | Pass rate below this triggers `LOW_PASS_RATE`    |
| `flagThresholds.latencyRisk` | 0.50    | Latency score below this triggers `LATENCY_RISK` |
| `flagThresholds.costRisk`    | 0.50    | Cost score below this triggers `COST_RISK`       |
| `evidenceLevel.strongMinN`   | 10      | Minimum test cases for "strong" evidence         |
| `evidenceLevel.mediumMinN`   | 5       | Minimum test cases for "medium" evidence         |

### Latency scoring

Latency is scored linearly between the good and bad bands:

* **Score 1.0** when `avgLatencyMs <= 1500ms`
* **Score 0.0** when `avgLatencyMs >= 8000ms`
* **Linear interpolation** between the two thresholds

If no latency data is available, the latency sub-score defaults to 1.0 (no penalty).

### Cost scoring

Cost is scored relative to the budget:

* **Score 1.0** when `avgCostUsd <= budgetUsd`
* **Linear decay** when cost exceeds budget
* **Score 0.0** when cost reaches 2x budget

If no cost or budget data is available, the cost sub-score defaults to 1.0 (no penalty).

### Custom metrics

You can add custom metrics to the score by providing `customMetrics` in the score inputs. Each custom metric has a `value` (0-1) and a `weight` (positive number). Custom metrics are blended with the base score proportionally to their total weight.

## Fallback behavior

When a data source is missing, the score uses conservative fallbacks:

| Missing data                | Fallback                     |
| --------------------------- | ---------------------------- |
| `safetyPassRate`            | Falls back to pass rate      |
| `judgeAvg`                  | Falls back to pass rate      |
| `schemaPassRate`            | Falls back to pass rate      |
| `avgLatencyMs`              | Defaults to 1.0 (no penalty) |
| `avgCostUsd` or `budgetUsd` | Defaults to 1.0 (no penalty) |

This means a run with only pass/fail data still gets a meaningful score, but the flags will indicate which dimensions are missing.

## Flags

Flags are warning indicators surfaced alongside the score. They don't change the score but alert you to data quality issues or risks:

| Flag                   | Trigger                              | Meaning                                                        |
| ---------------------- | ------------------------------------ | -------------------------------------------------------------- |
| `SAFETY_RISK`          | Safety score \< 0.95                 | Safety pass rate is below the acceptable threshold             |
| `LOW_PASS_RATE`        | Pass rate \< 0.90                    | Pass rate is below the acceptable threshold                    |
| `LATENCY_RISK`         | Latency score \< 0.50                | Average latency is in the poor range                           |
| `COST_RISK`            | Cost score \< 0.50                   | Average cost is significantly over budget                      |
| `MISSING_JUDGE`        | No judge data available              | LLM judge was not run or results not stored                    |
| `MISSING_SAFETY`       | No safety data available             | Safety assertions were not present                             |
| `LOW_N`                | Total cases \< 10                    | Sample size is too small for strong evidence                   |
| `SAFETY_WEAK_EVIDENCE` | Safety from keyword proxy            | Safety was inferred from error text, not structured assertions |
| `TRACE_COVERAGE_LOW`   | Trace coverage \< 0.80               | Less than 80% of trace-linked cases were matched               |
| `MISSING_PROVENANCE`   | No cost records for trace-linked run | Model/provider provenance is missing                           |

## Evidence levels

The evidence level indicates how trustworthy the score is based on data completeness:

| Level      | Requirements                                 | Meaning                                                      |
| ---------- | -------------------------------------------- | ------------------------------------------------------------ |
| **Strong** | Has judge data, has safety data, and n >= 10 | All major dimensions are covered with sufficient sample size |
| **Medium** | Has judge or safety data, and n >= 5         | Some dimensions covered with moderate sample size            |
| **Weak**   | Missing both judge and safety, or n \< 5     | Score is based primarily on pass rate with limited evidence  |

Use the evidence level to decide whether to trust a score for gating decisions. A "weak" score may still be useful for trend tracking but should not block a release on its own.

## Scoring spec versioning

Every quality score is persisted with:

* **`scoringVersion`** — the spec version used (currently `v1`)
* **`inputsJson`** — the exact inputs that produced the score
* **`scoringSpecJson`** — the full scoring spec at computation time
* **`inputsHash`** — SHA-256 hash of canonicalized inputs
* **`scoringSpecHash`** — SHA-256 hash of canonicalized scoring spec
* **`scoringCommit`** — Git commit SHA when the score was computed

This ensures scores are reproducible and auditable. You can verify the scoring formula by fetching `/api/quality/spec`, which returns the current scoring spec.

## API access

* **`GET /api/quality/spec`** — Returns the current scoring spec (weights, thresholds, version)
* **`GET /api/quality/{evaluationId}`** — Returns quality scores for an evaluation
* **`POST /api/quality/recompute/{runId}`** — Recomputes the quality score for a specific run