# EvalGate — AI Evaluation Platform

> One-command CI for AI evaluation. Complete pipeline: discover → manifest → impact → run → diff → PR summary. Published SDK on npm. REST API for any language.

## Machine-readable map

- [Homepage Markdown](https://www.evalgate.com/index.md)
- [Account creation requirements](https://www.evalgate.com/signup.md)
- [Pricing](https://www.evalgate.com/pricing.md)
- [Documentation index](https://www.evalgate.com/docs/llms.txt)
- [API index](https://www.evalgate.com/api/llms.txt)
- [OpenAPI 3.1](https://www.evalgate.com/openapi.json)
- [RFC 9727 API catalog](https://www.evalgate.com/.well-known/api-catalog)
- [ARD AI catalog](https://www.evalgate.com/.well-known/ai-catalog.json)
- [Agent Skill index](https://www.evalgate.com/.well-known/agent-skills/index.json)
- [Schema Map](https://www.evalgate.com/schemamap.xml)

## What is EvalGate?

EvalGate (AI Evaluation Platform) is a production-grade platform that helps developers build, test, and monitor AI systems with **one-command CI**. Think of it as a combination of:
- **Testing framework** for LLMs and agents (unit tests, LLM-as-judge, human eval, A/B tests)
- **Observability layer** for multi-agent workflows (traces, spans, DAG visualization)
- **Cost dashboard** for tracking LLM spend across providers
- **Governance engine** for compliance (SOC2, GDPR, HIPAA, etc.)
- **CI/CD integration** with GitHub Actions and smart regression detection

You sign up, create an organization, add API keys or use the web UI, then instrument your code with the SDK or call the REST API. Evaluations run test cases against your model outputs; workflows trace agent pipelines; costs are tracked per token.

## When to use EvalGate

Use EvalGate when you need to evaluate AI or agent behavior before release, collect traces from production workflows, compare changes with regression gates, or turn reviewed failures into repeatable CI coverage. Start locally with the SDK or CLI; use the REST API when integrating from another language. EvalGate is not an identity provider or a replacement for your model provider.

## How it works

1. **One-Command CI**: `evalgate ci` automatically discovers specs, runs impact analysis, executes evaluations, diffs results, and posts rich PR summaries
2. **Evaluations**: Create an evaluation (unit_test, human_eval, model_eval, ab_test), add test cases (input + expected output or rubric), then run it. The platform executes tests, scores outputs, and stores results. Use the visual builder or SDK.
3. **Workflows**: Instrument multi-agent pipelines with WorkflowTracer. Start a workflow, record agent spans, handoffs, decisions, and costs. View DAGs and run history in the dashboard.
4. **Traces**: Create traces and spans for any AI call. Attach metadata, record decisions, and track costs. Useful for debugging and auditing.
5. **Auth**: The web app supports Google/GitHub sign-in. API and MCP calls use a self-serve EvalGate API key in `Authorization: Bearer <key>`. EvalGate does not currently operate a third-party OAuth authorization server.

## Quick Start (60 seconds)

Add this to your `.github/workflows/evalgate.yml`:

```yaml
name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx @evalgate/sdk ci --format github --write-results --base main
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-results
          path: .evalgate/
```

Create `eval/your-spec.spec.ts`:

```typescript
import { defineEval } from "@evalgate/sdk";

defineEval({
  name: "Basic Math Operations",
  description: "Test fundamental arithmetic",
  prompt: "Test: 1+1=2, string concatenation, array includes",
  expected: "All tests should pass",
  tags: ["basic", "math"],
  category: "unit-test"
});
```

```bash
git add .github/workflows/evalgate.yml eval/
git commit -m "feat: add EvalGate CI pipeline"
git push
```

That's it! Your CI now discovers specs, runs only impacted code, diffs results, and posts rich summaries automatically.

## Install (SDK only)

npm install @evalgate/sdk
pip install evalgate-sdk

## CLI Commands

### 🚀 One-Command CI

```bash
npx @evalgate/sdk ci --format github --write-results --base main    # Complete CI pipeline
npx @evalgate/sdk ci --impacted-only                               # Run only impacted specs
npx @evalgate/sdk ci --format json                                  # JSON output for automation
```

### 🆕 v3.0.2 Features

#### Judge Credibility
```bash
npx @evalgate/sdk judge-credibility --labeled-dataset .evalgate/golden/labeled.jsonl
# Advanced TPR/TNR computation with Rogan-Gladen correction and bootstrap confidence intervals
```

#### Analyze Pipeline  
```bash
npx @evalgate/sdk analyze --labeled-dataset .evalgate/golden/labeled.jsonl
# Failure-modes taxonomy, impact ranking, and systematic AI quality analysis
```

#### Cost Tier API
```typescript
import { expect } from '@evalgate/sdk';

// Budget-controlled assertions
await expect(response).to.contain("Paris").withCostTier('medium');
await expect(code).to.beValid().withCostTier('code');
await expect(summary).to.beFactuallyConsistent().withCostTier('llm');
```

### Discovery & Analysis

```bash
npx @evalgate/sdk discover --manifest                               # Generate evaluation manifest
npx @evalgate/sdk impact-analysis --base main                       # Analyze impact of changes
npx @evalgate/sdk diff --base main --head last                       # Compare results
```

### Legacy Commands

```bash
npx @evalgate/sdk init              # Full scaffolder — detects repo, creates baseline, installs CI workflow
npx @evalgate/sdk gate              # Run regression gate (built-in or project mode)
npx @evalgate/sdk gate --format json # Machine-readable JSON output
npx @evalgate/sdk baseline update   # Re-run tests and update baseline
npx @evalgate/sdk upgrade --full    # Upgrade to full metric gate (golden eval, confidence, latency, cost)
npx @evalgate/sdk check             # Gate on quality score (requires API key)
npx @evalgate/sdk doctor            # Verify CI/CD setup
npx @evalgate/sdk label             # Interactive CLI for pass/fail + failure mode classification
npx @evalgate/sdk explain           # Root cause analysis with fix suggestions
```

## 🆕 Golden Path Documentation (v3.0.2)

EvalGate's **golden path** is the canonical 7-step evaluation loop that demonstrates the complete AI quality infrastructure. **Zero mocks, zero database, zero network required.**

### The 7-Step Loop
```
SDK payload → Validate → Freeze → Detect failures → Generate test case → Quarantine → Promote → PR annotation → Replay plan → Dataset health
```

### Quick Demo
```bash
# Run the complete golden path demo
pnpm tsx scripts/golden-path-demo.ts

# Run golden regression test
pnpm eval:golden
```

### Step-by-Step Breakdown

1. **SDK Payload Validation** - `validateTraceUpload()` checks specVersion and schema
2. **Freeze with Redaction** - `freezeTrace()` creates immutable snapshots with PII redaction
3. **Failure Detection** - `detectRuleBased()` identifies refusals, hallucinations, tool errors
4. **Test Case Generation** - `createGeneratedTestCase()` → `quarantineTestCase()` → `promoteTestCase()`
5. **PR Annotation** - `buildCheckRunPayload()` and `buildPRCommentBody()` for GitHub integration
6. **Replay Plan** - `buildReplayPlan()` pre-flights snapshots with tier filtering
7. **Dataset Health** - `analyzeDatasetHealth()` detects duplicates, outliers, and trends

### Golden Dataset Format
```json
{
  "name": "golden-regression",
  "minScore": 100,
  "baselineScore": 100,
  "cases": [
    { "input": "What is 2 + 2?", "expectedOutput": "4" },
    { "input": "What color is the sky?", "expectedOutput": "blue" },
    { "input": "Say 'ok' in one word.", "expectedOutput": "ok" }
  ]
}
```

### Labeled Dataset for Judge Credibility (v3.0.2)
```jsonl
{"caseId":"case-001","input":"What is 2+2?","expected":"4","actual":"4","label":"pass","failureMode":null,"labeledAt":"2026-03-09T09:00:00.000Z"}
{"caseId":"case-002","input":"Summarize this article","expected":"Brief summary","actual":"I cannot help","label":"fail","failureMode":"refusal","labeledAt":"2026-03-09T09:00:00.000Z"}
```

**Expected output**: All 7 steps execute with `✓` markers, demonstrating the complete reliability loop from production traces to regression gates.

## What EvalGate Provides

EvalGate provides:
- **One-Command CI**: Complete evaluation pipeline with smart caching and GitHub integration
- **Workflow Tracing**: Instrument multi-agent workflows with DAG visualization, agent handoffs, and decision auditing
- **Evaluation Engine**: 50+ evaluation templates, createTestSuite runner, 20 Expectation methods + 27 standalone assertion functions (21 sync + 6 LLM-backed async variants)
- **Cost Analytics**: Track LLM spending per model/provider with built-in pricing for OpenAI, Anthropic, Google (12+ models)
- **Agent Benchmarking**: Compare agent architectures (ReAct, CoT, ToT) across accuracy, latency, cost, and safety metrics
- **Governance Engine**: Enterprise-grade compliance with presets (SOC2, GDPR, HIPAA, FINRA_4511, PCI-DSS) and approval/blocking rules
- **Decision Auditing**: Record every agent decision with alternatives, confidence scores, and reasoning chains
- **Workflow Orchestration**: Multi-agent workflow management with DAG definitions and handoff tracking
- **Human Evaluation**: Annotation system for human evaluation tasks and collaborative review
- **Arena Competition**: AI model comparison platform with leaderboards and competition tracking
- **Interactive Playground**: Test your own AI outputs with assertions — no signup required
- **Published SDKs**: TypeScript/JavaScript SDK and Python SDK with zero-config init, auto-retry, caching, batching, and framework integrations

## SDK Usage

```typescript
import { AIEvalClient, WorkflowTracer } from '@evalgate/sdk';

const client = AIEvalClient.init(); // reads EVALGATE_API_KEY env var
const tracer = new WorkflowTracer(client);

await tracer.startWorkflow('My Pipeline', {
  nodes: [
    { id: 'router', type: 'agent', name: 'RouterAgent' },
    { id: 'worker', type: 'agent', name: 'WorkerAgent' },
  ],
  edges: [{ from: 'router', to: 'worker' }],
  entrypoint: 'router',
});

const span = await tracer.startAgentSpan('RouterAgent', { query: userInput });
await tracer.recordDecision({
  agent: 'RouterAgent',
  type: 'route',
  chosen: 'WorkerAgent',
  alternatives: [{ action: 'fallback', confidence: 20 }],
  confidence: 90,
});
await tracer.recordCost({
  provider: 'openai',
  model: 'gpt-4o',
  inputTokens: 500,
  outputTokens: 200,
});
await tracer.endAgentSpan(span, { result: 'routed' });
await tracer.endWorkflow({ status: 'success' });
```

## Evaluation with Assertions

```typescript
import { createTestSuite, expect } from '@evalgate/sdk';

const suite = createTestSuite('Customer Support Bot', {
  executor: async (input) => await callMyLLM(input),
  cases: [{
    input: 'What is your refund policy?',
    assertions: [
      (output) => expect(output).toContainKeywords(['refund', '30 days']),
      (output) => expect(output).toNotContainPII(),
      (output) => expect(output).toBeProfessional(),
    ]
  }]
});

const { total, passed, failed } = await suite.run();
```

## Python Usage (REST API)

```python
import requests, os

BASE = "https://evalgate.com"
HEADERS = {"Authorization": f"Bearer {os.environ['EVALGATE_API_KEY']}", "Content-Type": "application/json"}

# Create trace
trace = requests.post(f"{BASE}/api/traces", headers=HEADERS, json={
    "name": "Agent Run", "traceId": "run-001", "organizationId": 1, "status": "pending"
}).json()

# Record decision
requests.post(f"{BASE}/api/decisions", headers=HEADERS, json={
    "spanId": trace["id"], "agentName": "Router", "decisionType": "route",
    "chosen": "technical", "alternatives": [{"action": "billing", "confidence": 20}], "confidence": 85
})

# Record cost
requests.post(f"{BASE}/api/costs", headers=HEADERS, json={
    "spanId": trace["id"], "provider": "openai", "model": "gpt-4",
    "inputTokens": 1500, "outputTokens": 800
})
```

## Framework Integrations

```typescript
// LangChain
import { traceLangChainAgent } from '@evalgate/sdk';
const traced = traceLangChainAgent(executor, tracer);

// CrewAI
import { traceCrewAI } from '@evalgate/sdk';
const traced = traceCrewAI(crew, tracer);

// AutoGen
import { traceAutoGen } from '@evalgate/sdk';
const traced = traceAutoGen(conversation, tracer);
```

## Enhanced SDK Features

### Core Client & Tracing
- `AIEvalClient` — Main client with traces, evaluations, llmJudge, annotations, developer, organizations APIs
- `WorkflowTracer`, `createWorkflowTracer` — Multi-agent workflow instrumentation with DAG visualization
- `RequestCache`, `PaginatedIterator`, `RequestBatcher` — Performance utilities
- Context Propagation: `createContext`, `withContext`, `EvalContext` — Request context management

### Evaluation & Testing
- `createTestSuite` — Test suite builder with executor, cases, and assertion support
- `expect` — Assertion factory with 21 built-in methods
- 21 standalone assertion functions (`containsKeywords`, `matchesPattern`, `hasNoHallucinations`, `notContainsPII`, `hasNoToxicity`, `hasFactualAccuracy`, `hasReadabilityScore`, etc.)
- `batchProcess`, `streamEvaluation` — Streaming & batch processing

### Framework Integrations
- `traceOpenAI`, `traceAnthropic` — LLM provider wrappers
- `traceLangChainAgent`, `traceCrewAI`, `traceAutoGen` — Framework integrations

### Error Handling
- `EvalGateError`, `RateLimitError`, `AuthenticationError`, `NetworkError`, `SDKError` — Enhanced error hierarchy

### Governance (Server-side)
- `GovernanceEngine`, `CompliancePresets` — Governance rules (not in npm package, server-side only)

## Developer Tools & Analytics
- `GET /api/developer/api-keys` — Create and manage API keys
- `GET /api/developer/usage` — Usage analytics and billing portal
- `GET /api/developer/webhooks` — Webhook management and delivery tracking
- `GET /api/developer/usage/summary` — Usage summary and cost breakdown
- `GET /api/developer/api-keys/:id/usage` — API key usage tracking

## REST API Endpoints

All endpoints accept JSON. Authenticated endpoints require `Authorization: Bearer <key>` header.

### Traces
- `POST /api/traces` — Create trace (name, traceId, organizationId required)
- `GET /api/traces` — List traces (filter by organizationId, status, search)
- `GET /api/traces/:id` — Get single trace
- `DELETE /api/traces` — Delete traces
- `POST /api/traces/:id/spans` — Create span within a trace
- `GET /api/traces/:id/spans` — List spans for a trace

### Evaluations
- `POST /api/evaluations` — Create evaluation (name, type required)
- `GET /api/evaluations` — List evaluations
- `GET /api/evaluations/:id` — Get single evaluation
- `PATCH /api/evaluations/:id` — Update evaluation
- `DELETE /api/evaluations/:id` — Delete evaluation
- `POST /api/evaluations/:id/runs` — Create and execute an evaluation run
- `GET /api/evaluations/:id/runs` — List runs for an evaluation
- `GET /api/evaluations/:id/runs/:runId` — Get single run result
- `POST /api/evaluations/:id/test-cases` — Add test cases
- `GET /api/evaluations/:id/test-cases` — List test cases
- `DELETE /api/evaluations/:id/test-cases` — Delete test cases
- `POST /api/evaluations/:id/publish` — Publish evaluation
- `DELETE /api/evaluations/:id/publish` — Unpublish evaluation
- `GET /api/evaluation-templates` — List evaluation templates (public, no auth)

### LLM Judge
- `POST /api/llm-judge/evaluate` — Run LLM-as-judge evaluation
- `GET /api/llm-judge/configs` — List judge configurations
- `POST /api/llm-judge/configs` — Create judge config
- `GET /api/llm-judge/results` — List judge results
- `GET /api/llm-judge/alignment` — Get LLM-human alignment metrics

### Workflows
- `POST /api/workflows` — Create workflow with DAG definition
- `GET /api/workflows` — List workflows
- `GET /api/workflows/:id` — Get single workflow
- `PUT /api/workflows/:id` — Update workflow
- `DELETE /api/workflows/:id` — Delete workflow
- `POST /api/workflows/:id/runs` — Create workflow run
- `GET /api/workflows/:id/runs` — List workflow runs
- `POST /api/workflows/:id/handoffs` — Record agent handoff
- `GET /api/workflows/:id/handoffs` — List handoffs

### Annotations & Human Evaluation
- `GET /api/annotations` — List human evaluation annotations with filtering
- `GET /api/annotations/tasks` — Manage annotation tasks and assignments
- `GET /api/annotations/tasks/:id` — Get annotation task details
- `GET /api/annotations/tasks/:id/items` — List annotation items for a task
- `POST /api/annotations/tasks/:id/complete` — Complete annotation task

### Decisions & Costs
- `POST /api/decisions` — Record agent decision with alternatives
- `GET /api/decisions` — Query decisions by workflowRunId or spanId
- `GET /api/decisions/stats` — Decision statistics
- `POST /api/costs` — Record LLM cost (provider, model, tokens)
- `GET /api/costs` — Get cost breakdown or org summary
- `GET /api/costs/trends` — Daily cost trends with summary
- `GET /api/costs/pricing` — Get model pricing table
- `POST /api/costs/pricing` — Update model pricing

### Benchmarks
- `POST /api/benchmarks` — Create benchmark
- `GET /api/benchmarks` — List benchmarks
- `GET /api/benchmarks/:id` — Get benchmark details
- `DELETE /api/benchmarks/:id` — Delete benchmark
- `POST /api/benchmarks/:id/results` — Submit benchmark results
- `GET /api/benchmarks/:id/leaderboard` — Get benchmark leaderboard

### Arena & AI Competition
- `POST /api/arena-matches` — Create AI model competitions and comparisons
- `GET /api/arena/leaderboard` — Competition leaderboards and rankings
- `GET /api/arena-matches` — List arena matches and results
- `GET /api/arena-matches/:id` — Get match details and participant results
- `POST /api/arena-matches/:id/results` — Submit competition results

### Shadow Evaluations
- `GET /api/shadow-evals` — List shadow evaluations for A/B testing
- `GET /api/shadow-evals/:id` — Get shadow evaluation details
- `GET /api/shadow-evals/stats` — Shadow evaluation statistics and comparisons

### Reports & Quality
- `POST /api/reports` — Create signed report
- `GET /api/reports` — List org's shared reports
- `GET /api/quality` — Get quality score with trend analysis
- `POST /api/quality` — Recompute quality score for evaluation run
- `GET /api/quality/spec` — Get quality scoring specification
- `GET /api/report-cards` — Get evaluation report cards
- `GET /api/report-cards/:evaluationId` — Get report card for specific evaluation

### Organizations
- `GET /api/organizations` — List organizations (admin)
- `GET /api/organizations/current` — Get current user's organization

### Demo (public, no auth)
- `POST /api/demo/custom-eval` — Run assertions on custom AI input/output (rate-limited)
- `GET /api/demo/chatbot` — Chatbot demo data
- `GET /api/demo/rag` — RAG demo data
- `GET /api/demo/codegen` — Code generation demo data

## MCP and agent tools

EvalGate provides an experimental, authenticated MCP Streamable HTTP endpoint at `POST /api/mcp`. It uses the official MCP server package, is stateless, and exposes scope-filtered, read-only review tools. Send an EvalGate bearer key with the `eval:read` scope. Protected Resource Metadata is published at `/.well-known/oauth-protected-resource/api/mcp` only when a real authorization server is configured; the endpoint fails closed otherwise.

`GET /api/mcp/tools` and `POST /api/mcp/call` are the authenticated, proprietary EvalGate Tool API. They are not MCP transport endpoints and must not be used by standards-based MCP clients.

### Proprietary Tool API execution
- `eval.quality.latest` — Get latest quality score for an evaluation with baseline comparison
- `eval.get` — Get evaluation details including test cases and recent runs
- `eval.list` — List evaluations for the organization
- Mutation tools (`trace.create`, `trace.span.create`, `eval.testcase.add`, `eval.run`) are disabled by default and are advertised only when an operator explicitly enables the proprietary write tier.

## Agent onboarding

- Start without credentials at the rate-limited `POST /api/demo/custom-eval` endpoint or inspect public templates at `GET /api/evaluation-templates`.
- The first 10,000 completed test-case results each month are free.
- [Account creation](https://www.evalgate.com/signup.md) uses interactive GitHub or Google sign-in; it does not require an EvalGate password, a separate verification email, or a card for the free allowance.
- After web sign-in, create and revoke API keys yourself under Developer → API Keys; no sales contact is required. A machine agent cannot create an identity non-interactively.
- EvalGate does not currently advertise a separate production-like sandbox. Use the public demo only for credential-free evaluation and a normal free organization for authenticated integration testing.

### WebMCP Legacy Support
For legacy WebMCP compatibility:
- `list_evaluation_templates` — List evaluation templates by category (50+ templates). Public, no auth.
- `create_evaluation` — Create a new evaluation. Requires auth.
- `run_evaluation` — Execute an evaluation run. Requires auth.
- `get_evaluation_results` — Fetch run results with pass/fail metrics. Requires auth.
- `get_quality_score` — Get quality score for an evaluation. Requires auth.

### Background Jobs System
- `POST /api/jobs/run` — Cron endpoint for job processing (internal)
- `GET /api/jobs/dead` — Dead letter queue for failed jobs
- `POST /api/jobs/dead/bulk-retry` — Bulk retry dead jobs
- Job queue with retry logic, exponential backoff, and TTL-based reclamation

### Advanced Monitoring
- `GET /api/drift/alerts` — Performance regression alerts and drift detection
- `GET /api/metrics/slo` — Platform SLO metrics and performance monitoring
- `GET /api/audit-logs` — Admin audit trail for compliance
- `GET /api/stream` — Real-time SSE streaming for live updates

## Links

- npm: https://www.npmjs.com/package/@evalgate/sdk
- PyPI: https://pypi.org/project/evalgate-sdk/
- GitHub: https://github.com/evalgate
- Docs: https://www.evalgate.com/docs
- API Reference: https://www.evalgate.com/openapi.json
- Full agent reference: https://www.evalgate.com/llms-full.txt