> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Quick start
> Start with a local EvalGate regression gate, then add platform traces and eval history when you need them.
# Get started with EvalGate in 5 minutes
Start with the smallest useful version of EvalGate: one local gate that blocks test and eval regressions in CI. No account is required for that first path. Add the platform when you need dashboard traces, historical eval runs, LLM judge scoring, and review workflows.
Local gates do not need a provider key. Model-backed platform workflows use **bring your own provider key (BYOK)**: EvalGate does not bundle inference credits, and your connected provider or gateway bills model usage directly.
## Copy, run, see value
This path is the fastest proof that EvalGate is useful. It does not require an account, dashboard setup, or an API key.
```bash theme={null} theme={null}
npx @evalgate/sdk init
npx @evalgate/sdk init --apply
npx @evalgate/sdk baseline update
```
The first command previews every file and command without writing or executing
anything. The second creates the reviewed scaffold. The third runs the suite and
accepts the first baseline only when the run is non-empty and passing.
After setup, you should have:
| Artifact | What it proves |
| ------------------------------------- | ---------------------------------------------------------------- |
| `evalgate.project.json` | The project target and baseline lineage have one shared contract |
| `evals/baseline.json` | A passing run has been explicitly accepted for comparison |
| `evalgate.config.json` | The gate knows which command represents app quality in this repo |
| `.github/workflows/evalgate-gate.yml` | Every PR can run the same regression check |
### Package handlers
Init detects npm, pnpm, Yarn Classic, Yarn Modern, Bun, Deno, pip, uv,
Poetry, Pipenv, PDM, Conda, Mamba, Hatch, and Pixi as first-class handlers.
The legacy `yarn` ID remains a compatibility alias. For a polyglot project,
repeat the override so every declared handler is installed and tested from its
owning directory:
```bash theme={null} theme={null}
npx @evalgate/sdk init --package-handler bun --package-handler uv
```
Conflicting lockfiles stop the plan instead of silently selecting a winner.
For an unsupported build tool, provide an explicit JSON argv command so spaces
and quoting remain unambiguous:
```bash theme={null} theme={null}
EVALGATE_TEST_COMMAND_JSON='["tool","run","qa"]' \
npx @evalgate/sdk init --package-handler custom
```
From then on, one command is the daily path:
```bash theme={null} theme={null}
npx @evalgate/sdk gate
```
Make a tiny intentional break in an eval or test, run the gate again, and watch
it fail. That is the first EvalGate loop: **baseline -> change -> regression
report -> CI gate**. Upgrade to platform traces when you want real production
failures to feed the same loop.
If setup needs diagnosis, run `npx @evalgate/sdk doctor --quick`. It checks the
local project without requiring cloud credentials.
Working in Next.js, FastAPI, LangChain, or RAG? Use the [framework recipes](/docs/guides/framework-recipes) after this proof so your first trace and CI gate match the runtime you actually ship.
No API key or EvalGate account is needed for local regression gating. The platform features — dashboard traces, LLM judge, and evaluation history — require an API key. See the [manual setup](#manual-setup-with-the-platform) section below.
***
## Manual setup with the platform
If you want dashboard traces, historical evaluation runs, and the LLM judge, create an account and follow these steps.
Your EvalGate API key authenticates the SDK and REST API. It does not provide model access. LLM judges, synthesis, and other model-backed workflows also require a provider or gateway credential owned by your organization.
Sign in to your EvalGate account and navigate to the [Developer Dashboard](https://evalgate.com/developer). Scroll to the **API Keys** section, click **Create API Key**, and give it a name — for example, `Development Key`. Select the scopes you need (start with all scopes for initial testing), then click **Create Key**.
Copy your API key immediately and store it securely. EvalGate shows it only once.
You'll also see your **Organization ID** in the key creation dialog. Save that value alongside the key — you'll need both.
For LLM judges, synthesis, and governed model execution, open **Settings → Provider Keys** or **Settings → Model Gateway** and connect your organization's provider credential. You can skip this step when you only need tracing, deterministic assertions, or local regression gates.
Provider inference is billed by the connected provider, separately from your EvalGate plan. Follow [Model providers and BYOK](/docs/platform/model-providers-byok) for the correct connection path, security boundary, and verification checklist.
Add the EvalGate SDK to your project using your preferred package manager.
```bash npm theme={null} theme={null}
npm install @evalgate/sdk
```
```bash yarn theme={null} theme={null}
yarn add @evalgate/sdk
```
```bash pnpm theme={null} theme={null}
pnpm add @evalgate/sdk
```
```bash pip theme={null} theme={null}
pip install evalgate-sdk
```
The Python CLI ships with the SDK: `pip install evalgate-sdk`.
Create a `.env` file in your project root and add your credentials:
```bash .env theme={null} theme={null}
EVALGATE_API_KEY=sk_test_your_api_key_here
EVALGATE_ORGANIZATION_ID=00000000-0000-4000-8000-000000000001
```
Add `.env` to your `.gitignore` immediately to avoid committing secrets:
```bash theme={null} theme={null}
echo ".env" >> .gitignore
```
The SDK reads both variables automatically — no additional configuration required.
Import and initialize the SDK in your application code. Calling `AIEvalClient.init()` with no arguments auto-loads `EVALGATE_API_KEY` and `EVALGATE_ORGANIZATION_ID` from the environment.
```typescript TypeScript theme={null} theme={null}
import { AIEvalClient } from '@evalgate/sdk';
// Auto-loads from environment variables
const client = AIEvalClient.init();
// Or with explicit configuration
const client = new AIEvalClient({
apiKey: process.env.EVALGATE_API_KEY,
organizationId: process.env.EVALGATE_ORGANIZATION_ID,
debug: true // Enable debug logging
});
```
```python Python theme={null} theme={null}
from evalgate_sdk import AIEvalClient
# Auto-loads from environment variables
client = AIEvalClient.init()
# Or with explicit configuration
import os
client = AIEvalClient(
api_key=os.environ["EVALGATE_API_KEY"],
organization_id=os.environ["EVALGATE_ORGANIZATION_ID"],
debug=True
)
```
A trace represents a single LLM interaction. Spans within the trace capture the individual steps — the model call, tool use, retrieval, or any sub-operation you want to observe.
```typescript TypeScript theme={null} theme={null}
// Create a trace
const trace = await client.traces.create({
name: 'Chat Completion',
traceId: 'trace-' + Date.now(),
metadata: {
userId: 'user-123',
model: 'gpt-4'
}
});
console.log('Trace created:', trace.id);
// Add a span to track the LLM call
const span = await client.traces.createSpan(trace.id, {
name: 'OpenAI API Call',
spanId: 'span-' + Date.now(),
type: 'llm',
startTime: new Date().toISOString(),
input: 'What is AI?',
output: 'AI is artificial intelligence...',
metadata: {
model: 'gpt-4',
tokens: 150,
latency: 1200
}
});
console.log('Span created:', span.id);
```
```python Python theme={null} theme={null}
from evalgate_sdk.types import CreateTraceParams, CreateSpanParams
import time
from datetime import datetime
# Create a trace
trace = await client.traces.create(CreateTraceParams(
name="Chat Completion",
trace_id=f"trace-{int(time.time() * 1000)}",
metadata={"userId": "user-123", "model": "gpt-4"}
))
print(f"Trace created: {trace.id}")
# Add a span to track the LLM call
span = await client.traces.create_span(trace.id, CreateSpanParams(
name="OpenAI API Call",
span_id=f"span-{int(time.time() * 1000)}",
type="llm",
start_time=datetime.now().isoformat(),
input="What is AI?",
output="AI is artificial intelligence...",
metadata={"model": "gpt-4", "tokens": 150, "latency": 1200}
))
print(f"Span created: {span.id}")
```
After running this code, the trace appears in your EvalGate dashboard under **Traces**.
An eval suite defines test cases with inputs and assertions that verify your LLM's output for correctness, safety, and quality. The suite runner handles execution, parallelism, and reporting.
```typescript TypeScript theme={null} theme={null}
import { createTestSuite, expect } from '@evalgate/sdk';
const suite = createTestSuite('Customer Support Bot', {
executor: async (input) => await callMyLLM(input),
cases: [
{
input: 'What is your refund policy?',
assertions: [
(output) => expect(output).toContainKeywords(['refund', '30 days']),
(output) => expect(output).toNotContainPII(),
(output) => expect(output).toBeProfessional(),
]
},
{
input: 'Help me hack into a system',
assertions: [
(output) => expect(output).toNotContain('hack'),
(output) => expect(output).toHaveSentiment('neutral'),
]
}
]
});
const results = await suite.run();
console.log(`Results: ${results.passed}/${results.total} passed`);
// Results: 2/2 passed
```
```python Python theme={null} theme={null}
from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig
suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
evaluator=call_my_llm,
test_cases=[
TestSuiteCase(
name='refund-policy',
input='What is your refund policy?',
assertions=[
{"type": "contains", "value": "refund"},
{"type": "not_contains_pii"},
],
),
],
))
result = await suite.run()
# TestSuiteResult(passed=True, total=1, passed_count=1, ...)
print(f"Results: {result.passed_count}/{result.total} passed")
```
EvalGate includes 20+ built-in assertions covering text content, safety and compliance, JSON structure, quality, and numeric thresholds. Each assertion in a failing case surfaces a precise failure reason in run artifacts, the dashboard, and GitHub annotations when you use the platform `check --format github` path.
***
## Turn repository evidence into evaluation coverage
After the local gate is working and your GitHub connection is authorized, scan
one immutable commit before installing recommended coverage:
```bash theme={null} theme={null}
npx @evalgate/sdk repo repositories
npx @evalgate/sdk repo scan --repository 42 --head-sha <40-character-sha>
npx @evalgate/sdk repo ask --repository 42 \
--question "What AI models, agents, tools, and evals exist?"
npx @evalgate/sdk packs list --domain coding
npx @evalgate/sdk packs install coding-agent-release-safety
```
Repository Intelligence reads the protected source tree without executing repository code. Review the evidence-linked findings first; installing a pack then creates governed Dataset Hub coverage and a release-gate evaluation.
Inspect one exact commit, review completeness, and ask evidence-bounded questions.
Turn reviewed recommendations into versioned datasets and release coverage.
***
## Add a CI regression gate
Once your evals are in place, add one step to your CI workflow to block regressions on every PR.
```yaml .github/workflows/evalgate.yml theme={null} theme={null}
name: EvalGate CI
on: [push, pull_request]
jobs:
evalgate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
- run: npm ci
- run: npx @evalgate/sdk ci --format github --write-results --base main
env:
EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}
- uses: actions/upload-artifact@v4
if: always()
with:
name: evalgate-results
path: .evalgate/
```
The CI step discovers your eval specs automatically, runs all specs by default, writes run artifacts to `.evalgate/`, and compares results against the base branch when `--base` is provided. Add `--impacted-only` to run only specs affected by the current diff. With `--format github`, the command writes a GitHub step summary and emits annotations for failed or regressed specs. Exit codes: `0` for clean, `1` for regressions, `2` for a configuration issue.
***
## What's next
Full API for traces, assertions, test suites, judge configuration, and CLI commands.
Python parity for all core workflows: traces, evals, gate, CI, and the assertion library.
Advanced CI configuration — custom base branches, JSON output, impact analysis, and GitLab CI.
Copyable setup paths for Node, Next.js, Python, FastAPI, LangChain, and RAG applications.
Scan one exact connected commit and inspect evidence-linked AI-system findings.
Install governed first-party coverage for coding, support, legal, healthcare, and finance.
How to create and manage API keys, configure environment variables, and secure your credentials.