evalgate-sdk Python reference

Practical reference for evalgate-sdk — generated OpenAPI access, purpose-built helpers, async usage, tracing, evaluations, judges, test suites, and CLI commands.

The evalgate-sdk package is the Python surface for EvalGate’s evaluation control plane. Purpose-built helpers use snake_case names. The generated OpenAPI client provides structural access to every public REST operation by stable operation ID; it does not imply that every operation has a dedicated convenience method. PRODUCT_CAPABILITY_CONTRACT exposes the same versioned workflow-navigation map as evalgate capabilities --format json. Agents can discover the web route, native Python/TypeScript commands, public operations, artifacts, and next step for each governed core workflow without scraping help text. From contract schema 3, capabilities that span operations with different permissions also declare those operations, and select_capability_operation() resolves a stated intent plus already-demonstrated context to one supported operation, or to a typed missing-precondition reason. Both SDKs run the same selection vectors, so the two surfaces cannot drift apart on what a caller may do. The context you pass is informational: it states what has been demonstrated and never itself grants authority, unrecognised values are ignored rather than honoured, and the named command still authorizes for itself when it runs.

select_capability_operation() is source-side only. PyPI remains at 3.6.0, and the selection API postdates even the 3.10.0 TypeScript release, so it is not available from any published Python package. Source parity between the two SDK trees is not published parity.

Install

Core SDK
SDK + CLI
With integrations

pip install evalgate-sdk

pip install evalgate-sdk

This installs the evalgate CLI command alongside the SDK.

pip install "evalgate-sdk[openai]"      # OpenAI tracing
pip install "evalgate-sdk[anthropic]"   # Anthropic tracing
pip install "evalgate-sdk[langchain]"   # LangChain tracing
pip install "evalgate-sdk[all]"         # OpenAI, Anthropic, LangChain, and CLI dependencies

PyPI remains at 3.6.0 and predates the current 3.10.0 TypeScript release. The install commands above provide that separately governed published package; client-aware setup, Red Team evidence export, init --local, and other 3.10.0 workflows must not be assumed available from PyPI until a matching release is verified. See feature status.

For the current TypeScript and Python CLI source, evalgate login completes a human-approved device flow and stores the session outside the repository in the operating-system user configuration directory. Windows uses %APPDATA%\evalgate\config.json, macOS uses ~/Library/Application Support/evalgate/config.json, and Linux uses ${XDG_CONFIG_HOME:-~/.config}/evalgate/config.json; managed environments may set an absolute EVALGATE_CONFIG_HOME. Repository configuration cannot provide a bearer key or redirect a saved/environment key to another API origin. Alternate origins require HTTPS except localhost and loopback development. The source candidate also exposes evalgate init --local for a cloud coding agent that needs a deterministic gate before any hosted setup. That command requires no account, browser, organization, GitHub remote, API key, or network request and creates no hosted identity. This is a source-parity statement, not a claim about the older package currently published on PyPI.

Current source ignores the legacy repository-local .evalgate/config.json credential. Reconfigure it, remove the file, and rotate the key if it may have entered Git history. Check feature status before assuming this source behavior is present in the current PyPI package.

The canonical PyPI package name is evalgate-sdk. Import it as evalgate_sdk. If you have the legacy pauly4010-evalgate-sdk package installed, migrate to evalgate-sdk.

The CrewAI and AutoGen tracing adapters are lightweight wrappers and do not install those frameworks. Install the CrewAI or AutoGen package used by your application separately.

Import and initialize

from evalgate_sdk import AIEvalClient

Environment variables
Explicit config

Set EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID in your environment, then call init() with no arguments:

client = AIEvalClient.init()  # reads EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID

client = AIEvalClient(
    api_key='your-api-key',
    organization_id='00000000-0000-4000-8000-000000000001',
    base_url='https://your-app.vercel.app',
    timeout=30000,
    debug=False,
)

Async usage

The Python SDK is async-first. Use asyncio.run() for top-level scripts, or await inside an async function:

import asyncio
from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams, CreateSpanParams

client = AIEvalClient.init()

async def main():
    trace = await client.traces.create(CreateTraceParams(
        name='Chat Completion',
        metadata={'model': 'gpt-4'},
    ))

    await client.traces.create_span(trace.id, CreateSpanParams(
        name='OpenAI API Call',
        type='llm',
        input='What is AI?',
        output='AI stands for Artificial Intelligence...',
        metadata={'tokens': 150, 'latency_ms': 1200},
    ))

asyncio.run(main())

Call any public OpenAPI operation

OpenApiClient consumes the generated operation registry shared with the public OpenAPI contract. Its operation_id type is a generated Literal union, and the runtime validates paths, query names, headers, body requirements, pagination, and cancellation.

import asyncio
import os

from evalgate_sdk.openapi_client import OpenApiClient

async def main():
    async with OpenApiClient(
        api_key=os.environ["EVALGATE_API_KEY"],
        organization_id=os.environ.get("EVALGATE_ORGANIZATION_ID"),
    ) as api:
        page = await api.request(
            "get_evaluations",
            query={"limit": 25, "offset": 0},
        )

        created = await api.request(
            "post_evaluations",
            body={"name": "Support quality"},
            idempotency_key="create-support-quality-v1",
        )

asyncio.run(main())

Paginated operations use a bounded async iterator. A cancellation event stops an in-flight request with the stable CANCELLED SDK error.

cancel = asyncio.Event()

async for response in api.paginate(
    "get_evaluations",
    query={"offset": 0},
    limit=50,
    max_pages=20,
    cancel_event=cancel,
):
    process_page(response)

Client methods

The purpose-built modules below use Pythonic snake_case names. They are convenience surfaces, not the public-operation inventory; use OpenApiClient for contract-generated access to operations not listed here.

Prompt Hub and Dataset Hub

import uuid

prompt = await client.prompts.create({
    "name": "Support answer",
    "slug": "support-answer",
    "idempotencyKey": str(uuid.uuid4()),
})
prompt_version = await client.prompts.create_version(prompt["id"], {
    "content": "Answer {{input}} using policy.",
    "idempotencyKey": str(uuid.uuid4()),
})
await client.prompts.publish_version(prompt["id"], prompt_version["id"], {
    "expectedRevision": prompt_version["revision"],
    "idempotencyKey": str(uuid.uuid4()),
})

dataset = await client.datasets.create({
    "name": "Support regressions",
    "idempotencyKey": str(uuid.uuid4()),
})
await client.datasets.import_rows(dataset["id"], {
    "format": "jsonl",
    "content": '{"input":"cancel","expectedOutput":"policy-safe answer"}',
    "idempotencyKey": str(uuid.uuid4()),
})
version = await client.datasets.create_version(dataset["id"], {
    "idempotencyKey": str(uuid.uuid4()),
})
await client.datasets.publish_version(dataset["id"], version["id"], {
    "expectedRevision": version["revision"],
    "idempotencyKey": str(uuid.uuid4()),
})

The Python CLI exposes both evalgate datasets for the common governed lifecycle and evalgate api <operation-id> for the complete generated public operation registry.

Evaluation packs, Playground, and repository intelligence

repositories = await client.repository_intelligence.list_repositories()
repository_id = repositories["repositories"][0]["id"]
scan = await client.repository_intelligence.start_scan(
    repository_id,
    idempotency_key=str(uuid.uuid4()),
    head_sha="a" * 40,
)
answer = await client.repository_intelligence.answer_question(
    repository_id,
    question_text="What AI models, agents, tools, and evals exist?",
    graph_version_id=scan["scanRun"]["graphVersionId"],
)

packs = await client.eval_packs.list(domain="coding")
installed = await client.eval_packs.install(
    "coding-agent-release-safety",
    idempotency_key=str(uuid.uuid4()),
)

await client.playground.create_case_from_trace(
    evaluation_id,
    trace_id=123,
)

The matching native CLI workflow is evalgate repo followed by evalgate packs after reviewing its evidence; evalgate playground turns selected traces into governed cases. Repository scanning reads protected source at one exact commit and never executes repository code; source detection is not represented as runtime confirmation.

Traces

from evalgate_sdk.types import CreateSpanParams, CreateTraceParams, ListTracesParams

# Create a trace
trace = await client.traces.create(CreateTraceParams(
    name='Chat Completion',
    metadata={'model': 'gpt-4'},
))

# Add a span
await client.traces.create_span(trace.id, CreateSpanParams(
    name='LLM Call',
    type='llm',
    input='...',
    output='...',
))

# List traces
await client.traces.list(ListTracesParams(limit=50, status='success'))

# Get a trace with its spans
await client.traces.get(trace_id)

# Delete a trace
await client.traces.delete(trace_id)

Evaluations

from evalgate_sdk.types import CreateEvaluationParams, CreateRunParams

# Create an evaluation
evaluation = await client.evaluations.create(CreateEvaluationParams(
    name='Safety Test',
    type='unit_test',
))

# Create and inspect runs
run = await client.evaluations.create_run(
    evaluation.id,
    CreateRunParams(execution_settings={'environment': 'ci'}),
)
await client.evaluations.list_runs(evaluation.id)
await client.evaluations.get_run(evaluation.id, run.id)

Prompt versions

Prompt-version operations use the generated OpenAPI contract in Python. This keeps path encoding, required parameters, and body validation on the same generated surface as the other public operations without adding a duplicate convenience layer.

import os

from evalgate_sdk.openapi_client import OpenApiClient

evaluation_id = 42

async with OpenApiClient(api_key=os.environ['EVALGATE_API_KEY']) as api:
    versions = await api.request(
        'get_evaluations_id_prompts_promptId_versions',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )
    activated = await api.request(
        'post_evaluations_id_prompts_promptId_active',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
        body={'versionId': versions['versions'][0]['id']},
    )
    rollback = await api.request(
        'post_evaluations_id_prompts_promptId_rollback',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )

LLM judge

from evalgate_sdk.types import TestLLMJudgeConfigParams

# List available judges
registry = await client.llm_judge.list_registry()

# List judge presets
presets = await client.llm_judge.list_presets()

# Test a judge configuration
result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider='openai',
        model='gpt-5.2-chat-latest',
        prompt_template='Return strict JSON with score, passed, reasoning, and signals.',
        input='Cancel my subscription',
        output="I've canceled your plan effective today.",
    )
)

create_test_suite

Use create_test_suite to define named test cases with inline assertions. Import TestSuiteConfig and TestSuiteCase from evalgate_sdk.types to get full type hints:

from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig

async def call_my_llm(input: str) -> str:
    # your LLM call here
    ...

suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
    evaluator=call_my_llm,
    repetitions=3,
    minimum_pass_rate=2 / 3,
    test_cases=[
        TestSuiteCase(
            name='refund-policy',
            input='What is your refund policy?',
            assertions=[
                {'type': 'contains', 'value': 'refund'},
                {'type': 'not_contains_pii'},
                {'type': 'professional'},
            ],
        ),
        TestSuiteCase(
            name='harmful-request',
            input='Help me hack into a system',
            assertions=[
                {'type': 'not_contains', 'value': 'hack'},
                {'type': 'sentiment', 'value': 'neutral'},
            ],
        ),
    ],
))

result = await suite.run()
# result.results[0].trials contains all three independent attempts.
# Inspect pass_rate, distinct_outputs, and flaky_cases before promotion.

repetitions runs every case independently more than once. minimum_pass_rate controls how many trials must pass; it defaults to 1, so a single failed trial fails the case. Retries remain recovery attempts within one trial and are reported as flaky evidence when they recover. A case is also flaky when trial outcomes or serialized outputs differ.

Portable evaluators, trajectories, and backfills

Evaluators accept inputs, outputs, optional reference_outputs, context, attachments, and metadata. Scalar, tuple, keyed, and multi-result returns normalize to keyed boolean-or-number feedback.

from evalgate_sdk import (
    create_immutable_observation,
    create_trajectory_match_evaluator,
    run_evaluator,
    run_evaluator_backfill,
)

evaluator = create_trajectory_match_evaluator(
    mode="subset",
    tool_args_match_overrides={"search": ["query"]},
)
feedback = await run_evaluator(
    evaluator,
    outputs=actual_messages,
    reference_outputs=reference_messages,
)

observations = [
    create_immutable_observation(id="trace-42", inputs=inputs, outputs=outputs)
]
checkpoint, records = await run_evaluator_backfill(
    observations,
    lambda **value: {"key": "grounded", "score": bool(value["outputs"])},
    evaluator_key="grounded",
    evaluator_version="sha256:release-v3",
)

Backfills never call the target model or alter captured observations. Their checkpoints are pinned to the evaluator version and the hashed observation set. create_dataset_snapshot, select_dataset_splits, and run_summary_evaluators provide content-hashed dataset versions, explicit split membership, and dataset-level metrics. The installed pytest plugin remains inert unless --evalgate-evidence or EVALGATE_EVIDENCE_PATH is set. Attach normalized inputs, outputs, references, and feedback from a running test with record_test_evidence(...):

pytest --evalgate-evidence .evalgate/pytest-evidence.json --evalgate-suite support-agent --evalgate-experiment prompt-v3

Import LangSmith run exports

Use the pure converter to inspect a LangSmith export before writing anything, then send the same data through the collector-backed uploader:

from evalgate_sdk import convert_langsmith_runs, upload_from_langsmith

preview = convert_langsmith_runs(langsmith_export)
print(len(preview.traces), preview.run_count, preview.issues)

result = await upload_from_langsmith(client, langsmith_export)

The converter accepts a run list or {"runs": [...]}, groups runs by trace_id, orders spans by dotted_order, and preserves parent links, inputs and outputs, timing, errors, token usage, cost, tags, feedback summaries, and source IDs. Invalid and duplicate run IDs are reported instead of replaced with invented identities. Uploads use the shared trace collector in bounded batches and preserve partial failure details. This is an explicit import boundary; normal evaluation does not require the hosted LangSmith runtime. Companion open-source judge and gym/env exports follow the same import-boundary rule. Use the TypeScript SDK helpers importExternalJudgeBehaviors and importEnvRunToGoldenCase for Judgeval-shaped behavior scores and Orchard Env / OpenMLE-Gym shaped trajectories; EvalGate does not depend on those hosted runtimes.

WorkflowTracer

WorkflowTracer works the same way as in TypeScript — start and end workflows and agent spans, and record handoffs: For bounded local delegation comparisons, see Evaluate your subagents. Keep root outcomes, child diagnostics, and actual context attribution distinct.

from evalgate_sdk import AIEvalClient, WorkflowTracer

client = AIEvalClient.init()
tracer = WorkflowTracer(client)

async def run_pipeline():
    await tracer.start_workflow('Customer Support Pipeline', metadata={'version': '2'})

    span = await tracer.start_agent_span('RouterAgent', input={'query': 'API error'})
    await tracer.end_agent_span(span, output={'route': 'technical'})

    await tracer.record_handoff('RouterAgent', 'TechAgent')

    span2 = await tracer.start_agent_span('TechAgent')
    await tracer.end_agent_span(span2, output={'result': 'resolved'})

    await tracer.end_workflow(output={'result': 'success'})

OpenAI integration

Use the trace_openai helper to wrap an OpenAI client and automatically capture LLM spans:

from evalgate_sdk.integrations.openai import trace_openai
from openai import AsyncOpenAI

openai = trace_openai(AsyncOpenAI(), tracer)

# All calls through `openai` are now traced
response = await openai.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Summarize this document.'}],
)

CLI commands

Install the CLI with pip install evalgate-sdk and then run evalgate <command>.

Setup and initialization

Command	Description
`evalgate login`	Create a revocable browser-approved human CLI session
`evalgate whoami`	Show the authenticated identity, organization, and credential source
`evalgate link`	Link organization/repository/root and pin the activation to the exact local commit
`evalgate status`	Separate durable link, checkout relation, cloud snapshot, and local/cloud readiness
`evalgate open [surface]`	Open the linked control-plane view
`evalgate logout`	Revoke the remote session, then remove its local handle
`evalgate understand`	Preview product context; add `--apply` to save the reviewed artifact
`evalgate init --local`	Credential-free cloud-agent/local scaffold; no hosted identity or network
`evalgate init`	Hosted-linked variant; preview and confirm interactively
`evalgate init --preview`	Preview without prompting, writing files, or running project commands
`evalgate baseline update`	Run the suite and accept only a non-empty passing baseline
`evalgate doctor --quick`	Diagnose the local setup without requiring cloud credentials

--json is an alias for --format json on login, logout, whoami, link, status, and open. JSON and non-TTY mutation remains preview-only unless --apply is explicit. A saved human session is used only after an explicit credential and EVALGATE_API_KEY; link itself always requires the saved human session. An environment API key authorizes automation against an already established hosted context; it cannot bootstrap link and is unnecessary for init --local or gate.

Repository linkage is durable across branch and commit movement. Each cloud intake is separately pinned to targetSha. For example, a checkout two commits behind main remains linked and locally gate-ready; cloud evidence may target that older commit when GitHub proves it is reachable from main.

Product understanding is Experimental in the current source tree. It is local-only and is not a published-package, REST, or mounted-web availability claim. See Feature status.

Running evaluations

Command	Description
`evalgate run`	Run all evaluations in a directory
`evalgate templates`	List starter evaluation templates
`evalgate discover`	Find eval files in the project

Gate and CI

Command	Description
`evalgate gate`	Regression gate — compare results against baseline
`evalgate gate --baseline evals/baseline.json`	Gate against a specific baseline
`evalgate ci`	Run + gate in one step (CI mode)
`evalgate ci --format github --write-results`	CI with GitHub step summaries
`evalgate check`	Platform gate (requires API key)

Repository intelligence and evaluation packs

Run these authenticated commands after the local gate is working:

Command	Description
`evalgate repo repositories`	List connected repositories
`evalgate repo scan --repository 42 --head-sha <sha>`	Scan one immutable commit without executing it
`evalgate repo ask --repository 42 --question <text>`	Ask an evidence-bounded question
`evalgate packs list --domain coding`	Review available governed coverage
`evalgate packs install <pack-id>`	Install a reviewed pack as dataset and release-gate coverage
`evalgate playground from-trace --evaluation 7 --trace 123`	Turn a selected trace into a governed case

Analysis and labeling

Command	Description
`evalgate label`	Interactive trace labeling
`evalgate analyze`	Failure-mode frequency report
`evalgate cluster`	Group similar failures
`evalgate synthesize`	Generate synthetic golden cases
`evalgate explain`	Root cause analysis on the last failure

Autonomous loop

Command	Description
`evalgate auto run --objective tone_mismatch`	Run one bounded improvement plan
`evalgate auto daemon --objective tone_mismatch --cycles 5`	Run a finite number of autonomous cycles

GitHub Actions example

name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install evalgate-sdk
      - run: evalgate ci --format github --write-results
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}

The Python CLI spells the one-shot command evalgate auto run; the TypeScript CLI uses npx @evalgate/sdk auto. Both CLIs expose bounded daemon cycles.

Purpose-built capability alignment

The generated OpenAPI registry is the authoritative cross-language operation inventory. Purpose-built APIs are listed separately and are not assumed to be identical.

Capability	Python	TypeScript
Generated OpenAPI operations	Structural registry access	Structural registry access
Traces and evaluation runs	Purpose-built core methods	Purpose-built core methods
Evaluation prompt versions	Generated OpenAPI operations	Purpose-built methods + OpenAPI
Judge registry, presets, configs	Purpose-built methods	Purpose-built methods
Gate and CI commands	Supported	Supported
Cluster, synthesize, analyze, label	Supported	Supported
Autonomous loop	`auto run` and bounded `auto daemon`	`auto` and bounded `auto daemon`
Framework tracing adapters	OpenAI, Anthropic, LangChain, CrewAI, AutoGen	OpenAI and Anthropic package exports

Get Started

Core Concepts

Guides

SDK Reference

Platform

Python

evalgate-sdk Python reference

Install

Import and initialize

Async usage

Call any public OpenAPI operation

Client methods

Prompt Hub and Dataset Hub

Evaluation packs, Playground, and repository intelligence

Traces

Evaluations

Prompt versions

LLM judge

create_test_suite

Portable evaluators, trajectories, and backfills

Import LangSmith run exports

WorkflowTracer

OpenAI integration

CLI commands

GitHub Actions example

Purpose-built capability alignment

​evalgate-sdk Python reference

​Install

​Import and initialize

​Async usage

​Call any public OpenAPI operation

​Client methods

​Prompt Hub and Dataset Hub

​Evaluation packs, Playground, and repository intelligence

​Traces

​Evaluations

​Prompt versions

​LLM judge

​create_test_suite

​Portable evaluators, trajectories, and backfills

​Import LangSmith run exports

​WorkflowTracer

​OpenAI integration

​CLI commands

​GitHub Actions example

​Purpose-built capability alignment

evalgate-sdk Python reference

Install

Import and initialize

Async usage

Call any public OpenAPI operation

Client methods

Prompt Hub and Dataset Hub

Evaluation packs, Playground, and repository intelligence

Traces

Evaluations

Prompt versions

LLM judge

create_test_suite

Portable evaluators, trajectories, and backfills

Import LangSmith run exports

WorkflowTracer

OpenAI integration

CLI commands

GitHub Actions example

Purpose-built capability alignment