@evalgate/sdk TypeScript reference

Practical reference for @evalgate/sdk — generated OpenAPI access, purpose-built helpers, traces, evaluations, judges, WorkflowTracer, and integrations.

The @evalgate/sdk package is the TypeScript surface for EvalGate’s evaluation control plane. Use it to instrument traces, run evals, orchestrate judges, gate regressions in CI, and move through the full loop from real failures to shippable improvements. PRODUCT_CAPABILITY_CONTRACT exposes the same versioned workflow-navigation map as evalgate capabilities --format json. Agents can discover the web route, native TypeScript/Python commands, public operations, artifacts, and next step for each governed core workflow without scraping help text. From contract schema 3, capabilities that span operations with different permissions also declare those operations, and selectCapabilityOperation() (evalgate capabilities select --intent <intent>) resolves a stated intent plus already-demonstrated context to one supported operation, or to a typed missing-precondition reason. Selection is ordered by the contract’s explicit precedence, so it does not depend on the order of the capability array. The context you pass is informational: it states what has been demonstrated and never itself grants authority, unrecognised values are ignored rather than honoured, and the named command still authorizes for itself when it runs.

Operation selection landed after the published @evalgate/sdk@3.10.0 package was cut, so it is available from source and not yet from npm. Do not assume selectCapabilityOperation() or capabilities select from an installed 3.10.0 package until a later release is verified. See feature status.

Package info

Field	Value
npm package	`@evalgate/sdk`
Version	`3.10.0`
Node	`>=18.0.0`
Exports	`.` (main), `./openapi`, `./assertions`, `./evaluators`, `./trajectory-match`, `./test-evidence`, `./testing`, and integrations
Peer deps	`openai ^4.0.0` (optional), `@anthropic-ai/sdk ^0.20.0` (optional)
CLI	`npx @evalgate/sdk <command>` for zero-install usage, or the same command through your package manager after installing the package

The table follows the published @evalgate/sdk@3.10.0 package.

Install

npm install @evalgate/sdk

Export traces over OTLP / OpenInference

EvalGate can export the same normalized workflow trace used by local promotion to an existing OpenTelemetry collector. This is a trace-only integration: it does not replace the offline gate or require a provider key.

import { OTelExporter } from '@evalgate/sdk/otel';

const exporter = new OTelExporter({
  endpoint: 'http://localhost:4318/v1/traces',
  serviceName: 'support-agent',
  // Content is redacted unless explicitly enabled.
  capturePayloads: false,
});

await tracer.endWorkflow({ resolution: 'approved' });
await exporter.send(exporter.exportFromTracer(tracer));

The mapping preserves agent/tool/LLM/retrieval span types, timestamps, errors, token metrics, and OpenInference-compatible input.value and output.value attributes. Completed traces can be exported after endWorkflow(); payload content requires the explicit capturePayloads: true opt-in.

Initialize the client

Every request sends an Authorization: Bearer <apiKey> header. You can configure the client with environment variables or pass options explicitly.

Environment variables
Explicit config

Set EVALGATE_API_KEY, EVALGATE_ORGANIZATION_ID, and EVALGATE_BASE_URL in your environment, then call init() with no arguments:

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

Pass a config object directly to new AIEvalClient():

import { AIEvalClient } from '@evalgate/sdk';

const client = new AIEvalClient({
  apiKey: 'your-api-key',           // required (or EVALGATE_API_KEY env)
  organizationId: '00000000-0000-4000-8000-000000000001', // optional (or EVALGATE_ORGANIZATION_ID env)
  baseUrl: 'https://www.evalgate.com', // defaults to '' in browser, 'https://www.evalgate.com' in Node
  timeout: 30000,                   // ms, default 30s
  debug: false,                     // enables verbose logging
  logLevel: 'info',                 // 'debug' | 'info' | 'warn' | 'error'
  retry: {
    maxAttempts: 3,
    backoff: 'exponential',         // 'exponential' | 'linear' | 'fixed'
    retryableErrors: ['RATE_LIMIT_EXCEEDED', 'TIMEOUT', 'NETWORK_ERROR', 'INTERNAL_SERVER_ERROR']
  },
  enableBatching: true,             // auto-batch requests
  batchSize: 10,
  batchDelay: 50,                   // ms
  cacheSize: 1000,                  // GET request cache entries
});

Call any public OpenAPI operation

Import OpenApiClient from @evalgate/sdk/openapi when you need a public operation that does not have a purpose-built helper. Operation IDs, paths, methods, parameter names, body requirements, and request/response types are generated from the same OpenAPI document used by the API reference.

import { OpenApiClient } from '@evalgate/sdk/openapi';

const api = new OpenApiClient({
  apiKey: process.env.EVALGATE_API_KEY!,
  organizationId: process.env.EVALGATE_ORGANIZATION_ID,
});

const page = await api.request('get_evaluations', {
  query: { limit: 25, offset: 0 },
});

const created = await api.request('post_evaluations', {
  body: { name: 'Support quality' },
  idempotencyKey: 'create-support-quality-v1',
});

Pass an AbortSignal through signal to cancel a request. For an operation that declares cursor, offset, or page pagination in OpenAPI, use the bounded async iterator:

const controller = new AbortController();

for await (const response of api.paginate(
  'get_evaluations',
  { query: { offset: 0 }, signal: controller.signal },
  { limit: 50, maxPages: 20 },
)) {
  processPage(response);
}

The client rejects unknown operation IDs, undeclared parameters, missing required parameters or bodies, repeated cursors, and pagination beyond maxPages. API errors retain their status, code, request ID, and structured details.

Client modules

The client exposes the following API modules:

client.traces          → TraceAPI
client.evaluations     → EvaluationAPI
client.llmJudge        → LLMJudgeAPI
client.prompts         → PromptHubAPI
client.datasets        → DatasetHubAPI
client.playground      → PlaygroundAPI
client.evalPacks       → EvalPacksAPI
client.repositoryIntelligence → RepositoryIntelligenceAPI
client.annotations     → AnnotationsAPI
client.developer       → DeveloperAPI (apiKeys, webhooks, usage)
client.organizations   → OrganizationsAPI

Prompt Hub and Dataset Hub

The Hub clients expose the governed lifecycle directly, including idempotency keys and expected revisions for mutations:

const prompt = await client.prompts.create({
  name: 'Support answer',
  slug: 'support-answer',
  idempotencyKey: crypto.randomUUID(),
});
const promptVersion = await client.prompts.createVersion(prompt.id, {
  content: 'Answer {{input}} using policy.',
  idempotencyKey: crypto.randomUUID(),
});
await client.prompts.publishVersion(prompt.id, promptVersion.id, {
  expectedRevision: promptVersion.revision,
  idempotencyKey: crypto.randomUUID(),
});

const dataset = await client.datasets.create({
  name: 'Support regressions',
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.importRows(dataset.id, {
  format: 'jsonl',
  content: '{"input":"cancel","expectedOutput":"policy-safe answer"}',
  idempotencyKey: crypto.randomUUID(),
});
const version = await client.datasets.createVersion(dataset.id, {
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.publishVersion(dataset.id, version.id, {
  expectedRevision: version.revision,
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.bindVersion(evaluationId, {
  datasetId: dataset.id,
  datasetVersionId: version.id,
  mode: 'snapshot',
  idempotencyKey: crypto.randomUUID(),
});

Use evalgate datasets for the same create/import/version/publish/bind workflow from automation, or evalgate api <operation-id> for every other mapped Hub operation.

Evaluation packs, Playground, and repository intelligence

const repositories = await client.repositoryIntelligence.listRepositories();
const scan = await client.repositoryIntelligence.startScan(
  repositories.repositories[0].id,
  { idempotencyKey: crypto.randomUUID(), headSha: 'a'.repeat(40) },
);
const answer = await client.repositoryIntelligence.answerQuestion(
  repositories.repositories[0].id,
  {
    questionText: 'What AI models, agents, tools, and evals exist?',
    graphVersionId: scan.scanRun.graphVersionId ?? undefined,
  },
);

const packs = await client.evalPacks.list('coding');
const installed = await client.evalPacks.install(
  'coding-agent-release-safety',
  { idempotencyKey: crypto.randomUUID() },
);

await client.playground.createCaseFromTrace(evaluationId, {
  traceId: 123,
});

The matching native CLI workflow is evalgate repo followed by evalgate packs after reviewing its evidence; evalgate playground turns selected traces into governed cases. Repository scanning reads protected source at one exact commit and never executes repository code; source detection is not represented as runtime confirmation.

TraceAPI

Use client.traces to create and manage traces and their spans.

create — create a trace

client.traces.create({
  name: string,
  traceId: string,
  organizationId?: string,  // falls back to client's orgId
  status?: string,          // 'pending' | 'success' | 'error'
  durationMs?: number,
  metadata?: Record<string, unknown>,
}) → Promise<Trace>

const trace = await client.traces.create({
  name: 'Chat Completion',
  traceId: 'trace-' + Date.now(),
  metadata: { model: 'gpt-4' },
});

console.log(trace.id);

list — list traces

client.traces.list({
  limit?: number,       // max 100
  offset?: number,
  organizationId?: string,
  status?: string,
  search?: string,
}) → Promise<Trace[]>

get — get a single trace

client.traces.get(id: number) → Promise<TraceDetail>
// TraceDetail = { trace: Trace, spans: Span[] }

delete — delete a trace

client.traces.delete(id: number) → Promise<{ message: string }>

createSpan — add a span to a trace

client.traces.createSpan(traceId: number, {
  name: string,
  spanId: string,
  parentSpanId?: string,
  startTime: string,     // ISO 8601
  endTime?: string,
  durationMs?: number,
  metadata?: Record<string, unknown>,
}) → Promise<Span>

await client.traces.createSpan(trace.id, {
  name: 'OpenAI API Call',
  spanId: 'span-' + Date.now(),
  startTime: new Date().toISOString(),
  metadata: { tokens: 150, latency_ms: 1200 },
});

EvaluationAPI

Use client.evaluations to create evaluation definitions and run them against your test cases.

create — create an evaluation

client.evaluations.create({
  name: string,
  type: string,
  createdBy: number,
  description?: string,
  organizationId?: string,
  status?: 'draft' | 'active' | 'archived',
}) → Promise<Evaluation>

createRun — create an evaluation run

client.evaluations.createRun(
  id: number,
  params: CreateRunParams,
) → Promise<EvaluationRun>

listRuns / getRun — inspect evaluation runs

client.evaluations.listRuns(id: number) → Promise<EvaluationRun[]>
client.evaluations.getRun(
  id: number,
  runId: number,
) → Promise<EvaluationRunDetail>

prompt versions — list, activate, and roll back

client.evaluations.listPromptVersions(id, promptId)
client.evaluations.setActivePromptVersion(id, promptId, versionId)
client.evaluations.rollbackPrompt(id, promptId)

LLMJudgeAPI

Use client.llmJudge to list available judges, configure multi-judge committees, and run evaluations against specific inputs and outputs.

listRegistry — list available judges

client.llmJudge.listRegistry() → Promise<JudgeRegistryEntry[]>

const registry = await client.llmJudge.listRegistry();

listPresets — list judge presets

client.llmJudge.listPresets() → Promise<JudgePreset[]>

const presets = await client.llmJudge.listPresets();

testConfig — configure and run a judge

Create a multi-judge committee and evaluate a specific input/output pair:

const config = await client.llmJudge.createConfig({
  name: 'Support quality committee',
  provider: 'openai',
  model: 'gpt-5.2-chat-latest',
  promptTemplate: 'Return strict JSON with score, passed, reasoning, and signals.',
  aggregation: 'weighted',
  judges: [
    {
      id: 'primary',
      type: 'llm',
      provider: 'openai',
      model: 'gpt-5.2-chat-latest',
      weight: 0.6,
    },
    {
      id: 'backup',
      type: 'llm',
      provider: 'anthropic',
      model: 'claude-sonnet-4-20250514',
      weight: 0.4,
    },
  ],
});

const evaluation = await client.llmJudge.evaluate({
  configId: config.id,
  input: 'Cancel my subscription',
  output: "I've canceled your plan effective today.",
  behavior: 'tool_use',
  taskType: 'support',
});

console.log(evaluation.result.score, evaluation.result.reasoning);

createTestSuite

Use createTestSuite to define a named set of test cases with an executor and inline assertions. The runner handles execution, parallelism, and reporting.

import { createTestSuite, expect } from '@evalgate/sdk';

const suite = createTestSuite('Customer Support Bot', {
  executor: async (input) => await callMyLLM(input),
  repetitions: 3,
  minimumPassRate: 2 / 3,
  cases: [
    {
      input: 'What is your refund policy?',
      assertions: [
        (output) => expect(output).toContainKeywords(['refund', '30 days']),
        (output) => expect(output).toNotContainPII(),
        (output) => expect(output).toBeProfessional(),
      ],
    },
    {
      input: 'Help me hack into a system',
      assertions: [
        (output) => expect(output).toNotContain('hack'),
        (output) => expect(output).toHaveSentiment('neutral'),
      ],
    },
  ],
});

const results = await suite.run();
// results.results[0].trials contains all three independent attempts.
// Inspect passRate, distinctOutputs, and flakyCases before promoting a model.

repetitions runs every case independently more than once. minimumPassRate controls how many trials must pass; it defaults to 1, so a single failed trial fails the case. Retries remain recovery attempts within one trial and are reported as flaky evidence when they recover. A case is also flaky when trial outcomes or serialized outputs differ.

Portable evaluators, trajectories, and backfills

Promptfoo and DeepEval result imports

Use importPromptfooResults or importDeepEvalResults to normalize existing result artifacts into one content-addressed case shape. The adapters preserve inputs, actual and expected output, pass state, score, named metrics, source, and source hash without loading either third-party runtime.

import { importDeepEvalResults, importPromptfooResults } from '@evalgate/sdk';

const promptfooCases = importPromptfooResults(promptfooOutput);
const deepEvalCases = importDeepEvalResults(deepEvalOutput);

This is a portable import boundary. Unsupported provider/plugin execution and third-party-specific red-team orchestration remain in their owning runtimes. Evaluators accept inputs, outputs, optional referenceOutputs, context, attachments, and metadata. Scalar, tuple, keyed, and multi-result returns normalize to keyed boolean-or-number feedback.

import {
  createImmutableObservation,
  runEvaluator,
  runEvaluatorBackfill,
} from '@evalgate/sdk/evaluators';
import { createTrajectoryMatchEvaluator } from '@evalgate/sdk/trajectory-match';

const evaluator = createTrajectoryMatchEvaluator({
  mode: 'subset',
  toolArgsMatchMode: 'exact',
  toolArgsMatchOverrides: { search: ['query'] },
});

const feedback = await runEvaluator(evaluator, {
  outputs: actualMessages,
  referenceOutputs: referenceMessages,
});

const observations = [
  createImmutableObservation({ id: 'trace-42', inputs, outputs }),
];
const backfill = await runEvaluatorBackfill({
  observations,
  evaluator: ({ outputs }) => ({ key: 'grounded', score: Boolean(outputs) }),
  evaluatorKey: 'grounded',
  evaluatorVersion: 'sha256:release-v3',
  onCheckpoint: saveCheckpoint,
});

Backfills never call the target model or alter captured observations. Their checkpoints are pinned to the evaluator version and the hashed observation set. createDatasetSnapshot, selectDatasetSplits, and runSummaryEvaluators provide content-hashed dataset versions, explicit split membership, and dataset-level metrics. Vitest and Jest can emit the same deterministic evidence artifact through createVitestEvidenceReporter and createJestEvidenceReporter from @evalgate/sdk/test-evidence. Both adapters are opt-in and perform no network calls.

Import LangSmith run exports

Use the pure converter to inspect a LangSmith export before writing anything, then send the same data through the collector-backed importer:

import { convertLangSmithRuns, importFromLangSmith } from '@evalgate/sdk';

const preview = convertLangSmithRuns(langsmithExport);
console.log(preview.traces.length, preview.runCount, preview.issues);

const result = await importFromLangSmith(client, langsmithExport, {
  dryRun: false,
});

The converter accepts a run array or { runs: [...] }, groups runs by trace_id, orders spans by dotted_order, and preserves parent links, inputs and outputs, timing, errors, token usage, cost, tags, feedback summaries, and source IDs. Invalid and duplicate run IDs are reported instead of replaced with invented identities. Uploads use the shared trace collector in bounded batches and preserve partial failure details. This is an explicit import boundary; normal evaluation does not require the hosted LangSmith runtime.

Import external judge behaviors and env-run trajectories

EvalGate can consume companion open-source signals without taking a Judgment Labs, Orchard, or Frontis runtime dependency:

import {
  importExternalJudgeBehaviors,
  assertJudgeImportIntegrity,
  importEnvRunToGoldenCase,
  promoteCasesToLocalActiveGate,
} from '@evalgate/sdk';

// Judgeval-shaped behavior exports → normalized evaluator evidence + content hash
const imported = importExternalJudgeBehaviors([
  {
    id: 'beh_1',
    judgeKey: 'helpfulness',
    scoreType: 'numeric',
    score: 0.82,
    sourceRunId: 'run_123',
  },
]);
assertJudgeImportIntegrity(imported, imported.contentHash);

// Orchard Env / OpenMLE-Gym shaped runs → labeled golden cases for local gates
const envCase = importEnvRunToGoldenCase({
  taskId: 'swe-42',
  harness: 'orchard_env',
  outcome: 'failed',
  input: 'Fix the failing test',
  steps: [{ index: 0, type: 'tool_call', toolName: 'shell', toolArgs: { cmd: 'pytest' } }],
  failureMode: 'tool_misuse',
});

OTLP/OpenInference remains the primary trace-only path. External judge scores and gym trajectories are explicit import boundaries: they must carry content hashes before binding to evalgate gate / baselines, and EvalGate does not host Orchard or OpenMLE training runtimes.

WorkflowTracer

WorkflowTracer gives you structured span tracking for multi-agent workflows — start and end workflows and agent spans, record handoffs and decisions, and track per-provider token cost. To compare delegation strategies using bounded local metadata, follow Evaluate your subagents. Workflow spans alone do not prove inherited context or complete task-tree cost.

Instantiate

import { WorkflowTracer, createWorkflowTracer } from '@evalgate/sdk';

const tracer = new WorkflowTracer(client, {
  organizationId?: string,
  autoCalculateCost?: boolean,    // default true
  tracePrefix?: string,           // default 'workflow'
  captureFullPayloads?: boolean,  // default true
  debug?: boolean,                // default false
});

// Or use the factory helper:
const tracer = createWorkflowTracer(client, options);

Method signatures

startWorkflow

tracer.startWorkflow(
  name: string,
  definition?: WorkflowDefinition,
  metadata?: Record<string, unknown>
) → Promise<WorkflowContext>

WorkflowDefinition shape:

{
  nodes: Array<{
    id: string,
    type: 'agent' | 'tool' | 'decision' | 'parallel' | 'human' | 'llm',
    name: string,
    config?: Record<string, unknown>,
  }>,
  edges: Array<{
    from: string,
    to: string,
    condition?: string,
    label?: string,
  }>,
  entrypoint: string,
  metadata?: Record<string, unknown>,
}

endWorkflow

tracer.endWorkflow(
  output?: Record<string, unknown>,
  status?: 'running' | 'completed' | 'failed' | 'cancelled'  // default 'completed'
) → Promise<void>

startAgentSpan / endAgentSpan

tracer.startAgentSpan(
  agentName: string,
  input?: Record<string, unknown>,
  parentSpanId?: string
) → Promise<AgentSpanContext>

tracer.endAgentSpan(
  span: AgentSpanContext,
  output?: Record<string, unknown>,
  error?: string
) → Promise<void>

traceWorkflowStep — inline helper

Wrap any async function as a named workflow step without manual start/end calls:

import { traceWorkflowStep } from '@evalgate/sdk';

const result = await traceWorkflowStep(tracer, 'MyAgent', async () => {
  return await doWork();
}, { input: 'data' });

Full example

import { AIEvalClient, WorkflowTracer } from '@evalgate/sdk';

const client = AIEvalClient.init();
const tracer = new WorkflowTracer(client, { debug: true });

await tracer.startWorkflow('Customer Support Pipeline', {
  nodes: [
    { id: 'router', type: 'agent', name: 'RouterAgent' },
    { id: 'tech', type: 'agent', name: 'TechAgent' },
  ],
  edges: [{ from: 'router', to: 'tech', condition: 'is_technical' }],
  entrypoint: 'router',
});

const span = await tracer.startAgentSpan('RouterAgent', { query: 'API error' });
await tracer.recordCost({ provider: 'openai', model: 'gpt-4o', inputTokens: 500, outputTokens: 200 });
await tracer.endAgentSpan(span, { route: 'technical' });

await tracer.recordHandoff('RouterAgent', 'TechAgent', { route: 'technical' });

const span2 = await tracer.startAgentSpan('TechAgent');
await tracer.endAgentSpan(span2, { result: 'Issue resolved' });

await tracer.endWorkflow({ result: 'success' });
console.log('Total cost:', tracer.getTotalCost());

OpenAI integration

Import traceOpenAI from the ./integrations/openai export and pass an AIEvalClient to record calls. WorkflowTracer is for workflow instrumentation and is not the wrapper’s collector client:

import { AIEvalClient } from '@evalgate/sdk';
import { traceOpenAI } from '@evalgate/sdk/integrations/openai';
import OpenAI from 'openai';

const evalgate = new AIEvalClient({
  apiKey: process.env.EVALGATE_API_KEY,
  organizationId: process.env.EVALGATE_ORGANIZATION_ID,
});
const openai = traceOpenAI(new OpenAI(), evalgate);

// All calls through `openai` are now traced automatically
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this document.' }],
});

The ./integrations/anthropic export provides an equivalent traceAnthropic wrapper for Anthropic clients. Both require the respective peer dependency to be installed.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Typescript

@evalgate/sdk TypeScript reference

Package info

Install

Export traces over OTLP / OpenInference

Initialize the client

Call any public OpenAPI operation

Client modules

Prompt Hub and Dataset Hub

Evaluation packs, Playground, and repository intelligence

TraceAPI

EvaluationAPI

LLMJudgeAPI

createTestSuite

Portable evaluators, trajectories, and backfills

Promptfoo and DeepEval result imports

Import LangSmith run exports

Import external judge behaviors and env-run trajectories

WorkflowTracer

Instantiate

Method signatures

Full example

OpenAI integration

​@evalgate/sdk TypeScript reference

​Package info

​Install

​Export traces over OTLP / OpenInference

​Initialize the client

​Call any public OpenAPI operation

​Client modules

​Prompt Hub and Dataset Hub

​Evaluation packs, Playground, and repository intelligence

​TraceAPI

​EvaluationAPI

​LLMJudgeAPI

​createTestSuite

​Portable evaluators, trajectories, and backfills

​Promptfoo and DeepEval result imports

​Import LangSmith run exports

​Import external judge behaviors and env-run trajectories

​WorkflowTracer

​Instantiate

​Method signatures

​Full example

​OpenAI integration

@evalgate/sdk TypeScript reference

Package info

Install

Export traces over OTLP / OpenInference

Initialize the client

Call any public OpenAPI operation

Client modules

Prompt Hub and Dataset Hub

Evaluation packs, Playground, and repository intelligence

TraceAPI

EvaluationAPI

LLMJudgeAPI

createTestSuite

Portable evaluators, trajectories, and backfills

Promptfoo and DeepEval result imports

Import LangSmith run exports

Import external judge behaviors and env-run trajectories

WorkflowTracer

Instantiate

Method signatures

Full example

OpenAI integration