> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Typescript

# @evalgate/sdk TypeScript reference

> Practical reference for @evalgate/sdk — generated OpenAPI access, purpose-built helpers, traces, evaluations, judges, WorkflowTracer, and integrations.

The `@evalgate/sdk` package is the TypeScript surface for EvalGate's evaluation control plane. Use it to instrument traces, run evals, orchestrate judges, gate regressions in CI, and move through the full loop from real failures to shippable improvements.

`PRODUCT_CAPABILITY_CONTRACT` exposes the same versioned workflow-navigation
map as `evalgate capabilities --format json`. Agents can discover the web route,
native TypeScript/Python commands, public operations, artifacts, and next step
for each governed core workflow without scraping help text.

## Package info

| Field       | Value                                                                                                                               |
| ----------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| npm package | `@evalgate/sdk`                                                                                                                     |
| Version     | `3.7.4`                                                                                                                             |
| Node        | `>=18.0.0`                                                                                                                          |
| Exports     | `.` (main), `./openapi`, `./assertions`, `./testing`, `./integrations/openai`, `./integrations/anthropic`                           |
| Peer deps   | `openai ^4.0.0` (optional), `@anthropic-ai/sdk ^0.20.0` (optional)                                                                  |
| CLI         | `npx @evalgate/sdk <command>` for zero-install usage, or the same command through your package manager after installing the package |

## Install

```bash theme={null} theme={null}
npm install @evalgate/sdk
```

## Initialize the client

Every request sends an `Authorization: Bearer <apiKey>` header. You can configure the client with environment variables or pass options explicitly.

<Tabs>
  <Tab title="Environment variables">
    Set `EVALGATE_API_KEY`, `EVALGATE_ORGANIZATION_ID`, and `EVALGATE_BASE_URL` in your environment, then call `init()` with no arguments:

    ```typescript theme={null} theme={null}
    import { AIEvalClient } from '@evalgate/sdk';

    const client = AIEvalClient.init();
    ```
  </Tab>

  <Tab title="Explicit config">
    Pass a config object directly to `new AIEvalClient()`:

    ```typescript theme={null} theme={null}
    import { AIEvalClient } from '@evalgate/sdk';

    const client = new AIEvalClient({
      apiKey: 'your-api-key',           // required (or EVALGATE_API_KEY env)
      organizationId: '00000000-0000-4000-8000-000000000001', // optional (or EVALGATE_ORGANIZATION_ID env)
      baseUrl: 'https://api.evalgate.com', // defaults to '' in browser, 'https://api.evalgate.com' in Node
      timeout: 30000,                   // ms, default 30s
      debug: false,                     // enables verbose logging
      logLevel: 'info',                 // 'debug' | 'info' | 'warn' | 'error'
      retry: {
        maxAttempts: 3,
        backoff: 'exponential',         // 'exponential' | 'linear' | 'fixed'
        retryableErrors: ['RATE_LIMIT_EXCEEDED', 'TIMEOUT', 'NETWORK_ERROR', 'INTERNAL_SERVER_ERROR']
      },
      enableBatching: true,             // auto-batch requests
      batchSize: 10,
      batchDelay: 50,                   // ms
      cacheSize: 1000,                  // GET request cache entries
    });
    ```
  </Tab>
</Tabs>

## Call any public OpenAPI operation

Import `OpenApiClient` from `@evalgate/sdk/openapi` when you need a public operation that does not have a purpose-built helper. Operation IDs, paths, methods, parameter names, body requirements, and request/response types are generated from the same OpenAPI document used by the API reference.

```typescript theme={null} theme={null}
import { OpenApiClient } from '@evalgate/sdk/openapi';

const api = new OpenApiClient({
  apiKey: process.env.EVALGATE_API_KEY!,
  organizationId: process.env.EVALGATE_ORGANIZATION_ID,
});

const page = await api.request('get_evaluations', {
  query: { limit: 25, offset: 0 },
});

const created = await api.request('post_evaluations', {
  body: { name: 'Support quality' },
  idempotencyKey: 'create-support-quality-v1',
});
```

Pass an `AbortSignal` through `signal` to cancel a request. For an operation that declares `cursor`, `offset`, or `page` pagination in OpenAPI, use the bounded async iterator:

```typescript theme={null} theme={null}
const controller = new AbortController();

for await (const response of api.paginate(
  'get_evaluations',
  { query: { offset: 0 }, signal: controller.signal },
  { limit: 50, maxPages: 20 },
)) {
  processPage(response);
}
```

The client rejects unknown operation IDs, undeclared parameters, missing required parameters or bodies, repeated cursors, and pagination beyond `maxPages`. API errors retain their status, code, request ID, and structured details.

## Client modules

The client exposes the following API modules:

```
client.traces          → TraceAPI
client.evaluations     → EvaluationAPI
client.llmJudge        → LLMJudgeAPI
client.prompts         → PromptHubAPI
client.datasets        → DatasetHubAPI
client.playground      → PlaygroundAPI
client.evalPacks       → EvalPacksAPI
client.repositoryIntelligence → RepositoryIntelligenceAPI
client.annotations     → AnnotationsAPI
client.developer       → DeveloperAPI (apiKeys, webhooks, usage)
client.organizations   → OrganizationsAPI
```

### Prompt Hub and Dataset Hub

The Hub clients expose the governed lifecycle directly, including idempotency
keys and expected revisions for mutations:

```typescript theme={null} theme={null}
const prompt = await client.prompts.create({
  name: 'Support answer',
  slug: 'support-answer',
  idempotencyKey: crypto.randomUUID(),
});
const promptVersion = await client.prompts.createVersion(prompt.id, {
  content: 'Answer {{input}} using policy.',
  idempotencyKey: crypto.randomUUID(),
});
await client.prompts.publishVersion(prompt.id, promptVersion.id, {
  expectedRevision: promptVersion.revision,
  idempotencyKey: crypto.randomUUID(),
});

const dataset = await client.datasets.create({
  name: 'Support regressions',
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.importRows(dataset.id, {
  format: 'jsonl',
  content: '{"input":"cancel","expectedOutput":"policy-safe answer"}',
  idempotencyKey: crypto.randomUUID(),
});
const version = await client.datasets.createVersion(dataset.id, {
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.publishVersion(dataset.id, version.id, {
  expectedRevision: version.revision,
  idempotencyKey: crypto.randomUUID(),
});
await client.datasets.bindVersion(evaluationId, {
  datasetId: dataset.id,
  datasetVersionId: version.id,
  mode: 'snapshot',
  idempotencyKey: crypto.randomUUID(),
});
```

Use `evalgate datasets` for the same create/import/version/publish/bind workflow
from automation, or `evalgate api <operation-id>` for every other mapped Hub
operation.

### Evaluation packs, Playground, and repository intelligence

```typescript theme={null} theme={null}
const repositories = await client.repositoryIntelligence.listRepositories();
const scan = await client.repositoryIntelligence.startScan(
  repositories.repositories[0].id,
  { idempotencyKey: crypto.randomUUID(), headSha: 'a'.repeat(40) },
);
const answer = await client.repositoryIntelligence.answerQuestion(
  repositories.repositories[0].id,
  {
    questionText: 'What AI models, agents, tools, and evals exist?',
    graphVersionId: scan.scanRun.graphVersionId ?? undefined,
  },
);

const packs = await client.evalPacks.list('coding');
const installed = await client.evalPacks.install(
  'coding-agent-release-safety',
  { idempotencyKey: crypto.randomUUID() },
);

await client.playground.createCaseFromTrace(evaluationId, {
  traceId: 123,
});
```

The matching native CLI workflow is `evalgate repo` followed by `evalgate packs`
after reviewing its evidence; `evalgate playground` turns selected traces into
governed cases. Repository scanning reads protected source at one exact
commit and never executes repository code; source detection is not represented
as runtime confirmation.

## TraceAPI

Use `client.traces` to create and manage traces and their spans.

<AccordionGroup>
  <Accordion title="create — create a trace">
    ```typescript theme={null} theme={null}
    client.traces.create({
      name: string,
      traceId: string,
      organizationId?: string,  // falls back to client's orgId
      status?: string,          // 'pending' | 'success' | 'error'
      durationMs?: number,
      metadata?: Record<string, unknown>,
    }) → Promise<Trace>
    ```

    ```typescript theme={null} theme={null}
    const trace = await client.traces.create({
      name: 'Chat Completion',
      traceId: 'trace-' + Date.now(),
      metadata: { model: 'gpt-4' },
    });

    console.log(trace.id);
    ```
  </Accordion>

  <Accordion title="list — list traces">
    ```typescript theme={null} theme={null}
    client.traces.list({
      limit?: number,       // max 100
      offset?: number,
      organizationId?: string,
      status?: string,
      search?: string,
    }) → Promise<Trace[]>
    ```
  </Accordion>

  <Accordion title="get — get a single trace">
    ```typescript theme={null} theme={null}
    client.traces.get(id: number) → Promise<TraceDetail>
    // TraceDetail = { trace: Trace, spans: Span[] }
    ```
  </Accordion>

  <Accordion title="delete — delete a trace">
    ```typescript theme={null} theme={null}
    client.traces.delete(id: number) → Promise<{ message: string }>
    ```
  </Accordion>

  <Accordion title="createSpan — add a span to a trace">
    ```typescript theme={null} theme={null}
    client.traces.createSpan(traceId: number, {
      name: string,
      spanId: string,
      parentSpanId?: string,
      startTime: string,     // ISO 8601
      endTime?: string,
      durationMs?: number,
      metadata?: Record<string, unknown>,
    }) → Promise<Span>
    ```

    ```typescript theme={null} theme={null}
    await client.traces.createSpan(trace.id, {
      name: 'OpenAI API Call',
      spanId: 'span-' + Date.now(),
      startTime: new Date().toISOString(),
      metadata: { tokens: 150, latency_ms: 1200 },
    });
    ```
  </Accordion>
</AccordionGroup>

## EvaluationAPI

Use `client.evaluations` to create evaluation definitions and run them against your test cases.

<AccordionGroup>
  <Accordion title="create — create an evaluation">
    ```typescript theme={null} theme={null}
    client.evaluations.create({
      name: string,
      type: string,
      createdBy: number,
      description?: string,
      organizationId?: string,
      status?: 'draft' | 'active' | 'archived',
    }) → Promise<Evaluation>
    ```
  </Accordion>

  <Accordion title="createRun — create an evaluation run">
    ```typescript theme={null} theme={null}
    client.evaluations.createRun(
      id: number,
      params: CreateRunParams,
    ) → Promise<EvaluationRun>
    ```
  </Accordion>

  <Accordion title="listRuns / getRun — inspect evaluation runs">
    ```typescript theme={null} theme={null}
    client.evaluations.listRuns(id: number) → Promise<EvaluationRun[]>
    client.evaluations.getRun(
      id: number,
      runId: number,
    ) → Promise<EvaluationRunDetail>
    ```
  </Accordion>

  <Accordion title="prompt versions — list, activate, and roll back">
    ```typescript theme={null} theme={null}
    client.evaluations.listPromptVersions(id, promptId)
    client.evaluations.setActivePromptVersion(id, promptId, versionId)
    client.evaluations.rollbackPrompt(id, promptId)
    ```
  </Accordion>
</AccordionGroup>

## LLMJudgeAPI

Use `client.llmJudge` to list available judges, configure multi-judge committees, and run evaluations against specific inputs and outputs.

<AccordionGroup>
  <Accordion title="listRegistry — list available judges">
    ```typescript theme={null} theme={null}
    client.llmJudge.listRegistry() → Promise<JudgeRegistryEntry[]>
    ```

    ```typescript theme={null} theme={null}
    const registry = await client.llmJudge.listRegistry();
    ```
  </Accordion>

  <Accordion title="listPresets — list judge presets">
    ```typescript theme={null} theme={null}
    client.llmJudge.listPresets() → Promise<JudgePreset[]>
    ```

    ```typescript theme={null} theme={null}
    const presets = await client.llmJudge.listPresets();
    ```
  </Accordion>

  <Accordion title="testConfig — configure and run a judge">
    Create a multi-judge committee and evaluate a specific input/output pair:

    ```typescript theme={null} theme={null}
    const config = await client.llmJudge.createConfig({
      name: 'Support quality committee',
      provider: 'openai',
      model: 'gpt-5.2-chat-latest',
      promptTemplate: 'Return strict JSON with score, passed, reasoning, and signals.',
      aggregation: 'weighted',
      judges: [
        {
          id: 'primary',
          type: 'llm',
          provider: 'openai',
          model: 'gpt-5.2-chat-latest',
          weight: 0.6,
        },
        {
          id: 'backup',
          type: 'llm',
          provider: 'anthropic',
          model: 'claude-sonnet-4-20250514',
          weight: 0.4,
        },
      ],
    });

    const evaluation = await client.llmJudge.evaluate({
      configId: config.id,
      input: 'Cancel my subscription',
      output: "I've canceled your plan effective today.",
      behavior: 'tool_use',
      taskType: 'support',
    });

    console.log(evaluation.result.score, evaluation.result.reasoning);
    ```
  </Accordion>
</AccordionGroup>

## createTestSuite

Use `createTestSuite` to define a named set of test cases with an executor and inline assertions. The runner handles execution, parallelism, and reporting.

```typescript theme={null} theme={null}
import { createTestSuite, expect } from '@evalgate/sdk';

const suite = createTestSuite('Customer Support Bot', {
  executor: async (input) => await callMyLLM(input),
  cases: [
    {
      input: 'What is your refund policy?',
      assertions: [
        (output) => expect(output).toContainKeywords(['refund', '30 days']),
        (output) => expect(output).toNotContainPII(),
        (output) => expect(output).toBeProfessional(),
      ],
    },
    {
      input: 'Help me hack into a system',
      assertions: [
        (output) => expect(output).toNotContain('hack'),
        (output) => expect(output).toHaveSentiment('neutral'),
      ],
    },
  ],
});

const results = await suite.run();
// { name: 'Customer Support Bot', total: 2, passed: 2, failed: 0, results: [...] }
```

## WorkflowTracer

`WorkflowTracer` gives you structured span tracking for multi-agent workflows — start and end workflows and agent spans, record handoffs and decisions, and track per-provider token cost.

### Instantiate

```typescript theme={null} theme={null}
import { WorkflowTracer, createWorkflowTracer } from '@evalgate/sdk';

const tracer = new WorkflowTracer(client, {
  organizationId?: string,
  autoCalculateCost?: boolean,    // default true
  tracePrefix?: string,           // default 'workflow'
  captureFullPayloads?: boolean,  // default true
  debug?: boolean,                // default false
});

// Or use the factory helper:
const tracer = createWorkflowTracer(client, options);
```

### Method signatures

<AccordionGroup>
  <Accordion title="startWorkflow">
    ```typescript theme={null} theme={null}
    tracer.startWorkflow(
      name: string,
      definition?: WorkflowDefinition,
      metadata?: Record<string, unknown>
    ) → Promise<WorkflowContext>
    ```

    `WorkflowDefinition` shape:

    ```typescript theme={null} theme={null}
    {
      nodes: Array<{
        id: string,
        type: 'agent' | 'tool' | 'decision' | 'parallel' | 'human' | 'llm',
        name: string,
        config?: Record<string, unknown>,
      }>,
      edges: Array<{
        from: string,
        to: string,
        condition?: string,
        label?: string,
      }>,
      entrypoint: string,
      metadata?: Record<string, unknown>,
    }
    ```
  </Accordion>

  <Accordion title="endWorkflow">
    ```typescript theme={null} theme={null}
    tracer.endWorkflow(
      output?: Record<string, unknown>,
      status?: 'running' | 'completed' | 'failed' | 'cancelled'  // default 'completed'
    ) → Promise<void>
    ```
  </Accordion>

  <Accordion title="startAgentSpan / endAgentSpan">
    ```typescript theme={null} theme={null}
    tracer.startAgentSpan(
      agentName: string,
      input?: Record<string, unknown>,
      parentSpanId?: string
    ) → Promise<AgentSpanContext>

    tracer.endAgentSpan(
      span: AgentSpanContext,
      output?: Record<string, unknown>,
      error?: string
    ) → Promise<void>
    ```
  </Accordion>

  <Accordion title="traceWorkflowStep — inline helper">
    Wrap any async function as a named workflow step without manual start/end calls:

    ```typescript theme={null} theme={null}
    import { traceWorkflowStep } from '@evalgate/sdk';

    const result = await traceWorkflowStep(tracer, 'MyAgent', async () => {
      return await doWork();
    }, { input: 'data' });
    ```
  </Accordion>
</AccordionGroup>

### Full example

```typescript theme={null} theme={null}
import { AIEvalClient, WorkflowTracer } from '@evalgate/sdk';

const client = AIEvalClient.init();
const tracer = new WorkflowTracer(client, { debug: true });

await tracer.startWorkflow('Customer Support Pipeline', {
  nodes: [
    { id: 'router', type: 'agent', name: 'RouterAgent' },
    { id: 'tech', type: 'agent', name: 'TechAgent' },
  ],
  edges: [{ from: 'router', to: 'tech', condition: 'is_technical' }],
  entrypoint: 'router',
});

const span = await tracer.startAgentSpan('RouterAgent', { query: 'API error' });
await tracer.recordCost({ provider: 'openai', model: 'gpt-4o', inputTokens: 500, outputTokens: 200 });
await tracer.endAgentSpan(span, { route: 'technical' });

await tracer.recordHandoff('RouterAgent', 'TechAgent', { route: 'technical' });

const span2 = await tracer.startAgentSpan('TechAgent');
await tracer.endAgentSpan(span2, { result: 'Issue resolved' });

await tracer.endWorkflow({ result: 'success' });
console.log('Total cost:', tracer.getTotalCost());
```

## OpenAI integration

Import `traceOpenAI` from the `./integrations/openai` export to wrap an OpenAI client and automatically capture LLM spans:

```typescript theme={null} theme={null}
import { traceOpenAI } from '@evalgate/sdk/integrations/openai';
import OpenAI from 'openai';

const openai = traceOpenAI(new OpenAI(), tracer);

// All calls through `openai` are now traced automatically
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: 'Summarize this document.' }],
});
```

<Note>
  The `./integrations/anthropic` export provides an equivalent `traceAnthropic` wrapper for Anthropic clients. Both require the respective peer dependency to be installed.
</Note>