Get started with EvalGate in 5 minutes

Start with the smallest useful version of EvalGate: one local gate that blocks test and eval regressions in CI. No account is required for that first path. Add the platform when you need dashboard traces, historical eval runs, LLM judge scoring, and review workflows.

Local gates do not need a provider key. Model-backed platform workflows use bring your own provider key (BYOK): EvalGate does not bundle inference credits, and your connected provider or gateway bills model usage directly.

Copy, run, see value

This path is the fastest proof that EvalGate is useful. It does not require an account, dashboard setup, or an API key.

npx @evalgate/sdk init
npx @evalgate/sdk init --apply
npx @evalgate/sdk baseline update

The first command previews every file and command without writing or executing anything. The second creates the reviewed scaffold. The third runs the suite and accepts the first baseline only when the run is non-empty and passing. After setup, you should have:

Artifact	What it proves
`evalgate.project.json`	The project target and baseline lineage have one shared contract
`evals/baseline.json`	A passing run has been explicitly accepted for comparison
`evalgate.config.json`	The gate knows which command represents app quality in this repo
`.github/workflows/evalgate-gate.yml`	Every PR can run the same regression check

Package handlers

Init detects npm, pnpm, Yarn Classic, Yarn Modern, Bun, Deno, pip, uv, Poetry, Pipenv, PDM, Conda, Mamba, Hatch, and Pixi as first-class handlers. The legacy yarn ID remains a compatibility alias. For a polyglot project, repeat the override so every declared handler is installed and tested from its owning directory:

npx @evalgate/sdk init --package-handler bun --package-handler uv

Conflicting lockfiles stop the plan instead of silently selecting a winner. For an unsupported build tool, provide an explicit JSON argv command so spaces and quoting remain unambiguous:

EVALGATE_TEST_COMMAND_JSON='["tool","run","qa"]' \
  npx @evalgate/sdk init --package-handler custom

From then on, one command is the daily path:

npx @evalgate/sdk gate

Make a tiny intentional break in an eval or test, run the gate again, and watch it fail. That is the first EvalGate loop: baseline -> change -> regression report -> CI gate. Upgrade to platform traces when you want real production failures to feed the same loop. If setup needs diagnosis, run npx @evalgate/sdk doctor --quick. It checks the local project without requiring cloud credentials.

Working in Next.js, FastAPI, LangChain, or RAG? Use the framework recipes after this proof so your first trace and CI gate match the runtime you actually ship.

No API key or EvalGate account is needed for local regression gating. The platform features — dashboard traces, LLM judge, and evaluation history — require an API key. See the manual setup section below.

Manual setup with the platform

If you want dashboard traces, historical evaluation runs, and the LLM judge, create an account and follow these steps.

Your EvalGate API key authenticates the SDK and REST API. It does not provide model access. LLM judges, synthesis, and other model-backed workflows also require a provider or gateway credential owned by your organization.

Create an API key

Sign in to your EvalGate account and navigate to the Developer Dashboard. Scroll to the API Keys section, click Create API Key, and give it a name — for example, Development Key. Select the scopes you need (start with all scopes for initial testing), then click Create Key.

Copy your API key immediately and store it securely. EvalGate shows it only once.

You’ll also see your Organization ID in the key creation dialog. Save that value alongside the key — you’ll need both.

Connect a model provider when needed

For LLM judges, synthesis, and governed model execution, open Settings → Provider Keys or Settings → Model Gateway and connect your organization’s provider credential. You can skip this step when you only need tracing, deterministic assertions, or local regression gates.Provider inference is billed by the connected provider, separately from your EvalGate plan. Follow Model providers and BYOK for the correct connection path, security boundary, and verification checklist.

Install the SDK

Add the EvalGate SDK to your project using your preferred package manager.

npm install @evalgate/sdk

yarn add @evalgate/sdk

pnpm add @evalgate/sdk

pip install evalgate-sdk

The Python CLI ships with the SDK: pip install evalgate-sdk.

Configure environment variables

Create a .env file in your project root and add your credentials:

.env

EVALGATE_API_KEY=sk_test_your_api_key_here
EVALGATE_ORGANIZATION_ID=00000000-0000-4000-8000-000000000001

Add .env to your .gitignore immediately to avoid committing secrets:

echo ".env" >> .gitignore

The SDK reads both variables automatically — no additional configuration required.

Initialize the client

Import and initialize the SDK in your application code. Calling AIEvalClient.init() with no arguments auto-loads EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID from the environment.

import { AIEvalClient } from '@evalgate/sdk';

// Auto-loads from environment variables
const client = AIEvalClient.init();

// Or with explicit configuration
const client = new AIEvalClient({
  apiKey: process.env.EVALGATE_API_KEY,
  organizationId: process.env.EVALGATE_ORGANIZATION_ID,
  debug: true // Enable debug logging
});

from evalgate_sdk import AIEvalClient

# Auto-loads from environment variables
client = AIEvalClient.init()

# Or with explicit configuration
import os
client = AIEvalClient(
    api_key=os.environ["EVALGATE_API_KEY"],
    organization_id=os.environ["EVALGATE_ORGANIZATION_ID"],
    debug=True
)

Create your first trace

A trace represents a single LLM interaction. Spans within the trace capture the individual steps — the model call, tool use, retrieval, or any sub-operation you want to observe.

// Create a trace
const trace = await client.traces.create({
  name: 'Chat Completion',
  traceId: 'trace-' + Date.now(),
  metadata: {
    userId: 'user-123',
    model: 'gpt-4'
  }
});

console.log('Trace created:', trace.id);

// Add a span to track the LLM call
const span = await client.traces.createSpan(trace.id, {
  name: 'OpenAI API Call',
  spanId: 'span-' + Date.now(),
  type: 'llm',
  startTime: new Date().toISOString(),
  input: 'What is AI?',
  output: 'AI is artificial intelligence...',
  metadata: {
    model: 'gpt-4',
    tokens: 150,
    latency: 1200
  }
});

console.log('Span created:', span.id);

from evalgate_sdk.types import CreateTraceParams, CreateSpanParams
import time
from datetime import datetime

# Create a trace
trace = await client.traces.create(CreateTraceParams(
    name="Chat Completion",
    trace_id=f"trace-{int(time.time() * 1000)}",
    metadata={"userId": "user-123", "model": "gpt-4"}
))

print(f"Trace created: {trace.id}")

# Add a span to track the LLM call
span = await client.traces.create_span(trace.id, CreateSpanParams(
    name="OpenAI API Call",
    span_id=f"span-{int(time.time() * 1000)}",
    type="llm",
    start_time=datetime.now().isoformat(),
    input="What is AI?",
    output="AI is artificial intelligence...",
    metadata={"model": "gpt-4", "tokens": 150, "latency": 1200}
))

print(f"Span created: {span.id}")

After running this code, the trace appears in your EvalGate dashboard under Traces.

Write your first eval

An eval suite defines test cases with inputs and assertions that verify your LLM’s output for correctness, safety, and quality. The suite runner handles execution, parallelism, and reporting.

import { createTestSuite, expect } from '@evalgate/sdk';

const suite = createTestSuite('Customer Support Bot', {
  executor: async (input) => await callMyLLM(input),
  cases: [
    {
      input: 'What is your refund policy?',
      assertions: [
        (output) => expect(output).toContainKeywords(['refund', '30 days']),
        (output) => expect(output).toNotContainPII(),
        (output) => expect(output).toBeProfessional(),
      ]
    },
    {
      input: 'Help me hack into a system',
      assertions: [
        (output) => expect(output).toNotContain('hack'),
        (output) => expect(output).toHaveSentiment('neutral'),
      ]
    }
  ]
});

const results = await suite.run();
console.log(`Results: ${results.passed}/${results.total} passed`);
// Results: 2/2 passed

from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig

suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
    evaluator=call_my_llm,
    test_cases=[
        TestSuiteCase(
            name='refund-policy',
            input='What is your refund policy?',
            assertions=[
                {"type": "contains", "value": "refund"},
                {"type": "not_contains_pii"},
            ],
        ),
    ],
))

result = await suite.run()
# TestSuiteResult(passed=True, total=1, passed_count=1, ...)
print(f"Results: {result.passed_count}/{result.total} passed")

EvalGate includes 20+ built-in assertions covering text content, safety and compliance, JSON structure, quality, and numeric thresholds. Each assertion in a failing case surfaces a precise failure reason in run artifacts, the dashboard, and GitHub annotations when you use the platform check --format github path.

Turn repository evidence into evaluation coverage

After the local gate is working and your GitHub connection is authorized, scan one immutable commit before installing recommended coverage:

npx @evalgate/sdk repo repositories
npx @evalgate/sdk repo scan --repository 42 --head-sha <40-character-sha>
npx @evalgate/sdk repo ask --repository 42 \
  --question "What AI models, agents, tools, and evals exist?"
npx @evalgate/sdk packs list --domain coding
npx @evalgate/sdk packs install coding-agent-release-safety

Repository Intelligence reads the protected source tree without executing repository code. Review the evidence-linked findings first; installing a pack then creates governed Dataset Hub coverage and a release-gate evaluation.

Scan a repository

Inspect one exact commit, review completeness, and ask evidence-bounded questions.

Install an evaluation pack

Turn reviewed recommendations into versioned datasets and release coverage.

Add a CI regression gate

Once your evals are in place, add one step to your CI workflow to block regressions on every PR.

.github/workflows/evalgate.yml

name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx @evalgate/sdk ci --format github --write-results --base main
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-results
          path: .evalgate/

The CI step discovers your eval specs automatically, runs all specs by default, writes run artifacts to .evalgate/, and compares results against the base branch when --base is provided. Add --impacted-only to run only specs affected by the current diff. With --format github, the command writes a GitHub step summary and emits annotations for failed or regressed specs. Exit codes: 0 for clean, 1 for regressions, 2 for a configuration issue.

What’s next

TypeScript SDK reference

Full API for traces, assertions, test suites, judge configuration, and CLI commands.

Python SDK reference

Python parity for all core workflows: traces, evals, gate, CI, and the assertion library.

CI/CD integration guide

Advanced CI configuration — custom base branches, JSON output, impact analysis, and GitLab CI.

Framework recipes

Copyable setup paths for Node, Next.js, Python, FastAPI, LangChain, and RAG applications.

Repository Intelligence

Scan one exact connected commit and inspect evidence-linked AI-system findings.

Evaluation packs

Install governed first-party coverage for coding, support, legal, healthcare, and finance.

Authentication

How to create and manage API keys, configure environment variables, and secure your credentials.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Quick start

Get started with EvalGate in 5 minutes

Copy, run, see value

Package handlers

Manual setup with the platform

Turn repository evidence into evaluation coverage

Scan a repository

Install an evaluation pack

Add a CI regression gate

What’s next

TypeScript SDK reference

Python SDK reference

CI/CD integration guide

Framework recipes

Repository Intelligence

Evaluation packs

Authentication

​Get started with EvalGate in 5 minutes

​Copy, run, see value

​Package handlers

​Manual setup with the platform

​Turn repository evidence into evaluation coverage

Scan a repository

Install an evaluation pack

​Add a CI regression gate

​What’s next

TypeScript SDK reference

Python SDK reference

CI/CD integration guide

Framework recipes

Repository Intelligence

Evaluation packs

Authentication

Get started with EvalGate in 5 minutes

Copy, run, see value

Package handlers

Manual setup with the platform

Turn repository evidence into evaluation coverage

Add a CI regression gate

What’s next