> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Python

# evalgate-sdk Python reference

> Practical reference for evalgate-sdk — generated OpenAPI access, purpose-built helpers, async usage, tracing, evaluations, judges, test suites, and CLI commands.

The `evalgate-sdk` package is the Python surface for EvalGate's evaluation control plane. Purpose-built helpers use snake\_case names. The generated OpenAPI client provides structural access to every public REST operation by stable operation ID; it does not imply that every operation has a dedicated convenience method.

`PRODUCT_CAPABILITY_CONTRACT` exposes the same versioned workflow-navigation
map as `evalgate capabilities --format json`. Agents can discover the web route,
native Python/TypeScript commands, public operations, artifacts, and next step
for each governed core workflow without scraping help text.

## Install

<Tabs>
  <Tab title="Core SDK">
    ```bash theme={null} theme={null}
    pip install evalgate-sdk
    ```
  </Tab>

  <Tab title="SDK + CLI">
    ```bash theme={null} theme={null}
    pip install evalgate-sdk
    ```

    This installs the `evalgate` CLI command alongside the SDK.
  </Tab>

  <Tab title="With integrations">
    ```bash theme={null} theme={null}
    pip install "evalgate-sdk[openai]"      # OpenAI tracing
    pip install "evalgate-sdk[anthropic]"   # Anthropic tracing
    pip install "evalgate-sdk[langchain]"   # LangChain tracing
    pip install "evalgate-sdk[all]"         # OpenAI, Anthropic, LangChain, and CLI dependencies
    ```
  </Tab>
</Tabs>

<Note>
  The canonical PyPI package name is `evalgate-sdk`. Import it as `evalgate_sdk`. If you have the legacy `pauly4010-evalgate-sdk` package installed, migrate to `evalgate-sdk`.
</Note>

<Note>
  The CrewAI and AutoGen tracing adapters are lightweight wrappers and do not install those frameworks. Install the CrewAI or AutoGen package used by your application separately.
</Note>

## Import and initialize

```python theme={null} theme={null}
from evalgate_sdk import AIEvalClient
```

<Tabs>
  <Tab title="Environment variables">
    Set `EVALGATE_API_KEY` and `EVALGATE_ORGANIZATION_ID` in your environment, then call `init()` with no arguments:

    ```python theme={null} theme={null}
    client = AIEvalClient.init()  # reads EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID
    ```
  </Tab>

  <Tab title="Explicit config">
    ```python theme={null} theme={null}
    client = AIEvalClient(
        api_key='your-api-key',
        organization_id='00000000-0000-4000-8000-000000000001',
        base_url='https://your-app.vercel.app',
        timeout=30000,
        debug=False,
    )
    ```
  </Tab>
</Tabs>

## Async usage

The Python SDK is async-first. Use `asyncio.run()` for top-level scripts, or `await` inside an async function:

```python theme={null} theme={null}
import asyncio
from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams, CreateSpanParams

client = AIEvalClient.init()

async def main():
    trace = await client.traces.create(CreateTraceParams(
        name='Chat Completion',
        metadata={'model': 'gpt-4'},
    ))

    await client.traces.create_span(trace.id, CreateSpanParams(
        name='OpenAI API Call',
        type='llm',
        input='What is AI?',
        output='AI stands for Artificial Intelligence...',
        metadata={'tokens': 150, 'latency_ms': 1200},
    ))

asyncio.run(main())
```

## Call any public OpenAPI operation

`OpenApiClient` consumes the generated operation registry shared with the public OpenAPI contract. Its `operation_id` type is a generated `Literal` union, and the runtime validates paths, query names, headers, body requirements, pagination, and cancellation.

```python theme={null} theme={null}
import asyncio
import os

from evalgate_sdk.openapi_client import OpenApiClient

async def main():
    async with OpenApiClient(
        api_key=os.environ["EVALGATE_API_KEY"],
        organization_id=os.environ.get("EVALGATE_ORGANIZATION_ID"),
    ) as api:
        page = await api.request(
            "get_evaluations",
            query={"limit": 25, "offset": 0},
        )

        created = await api.request(
            "post_evaluations",
            body={"name": "Support quality"},
            idempotency_key="create-support-quality-v1",
        )

asyncio.run(main())
```

Paginated operations use a bounded async iterator. A cancellation event stops an in-flight request with the stable `CANCELLED` SDK error.

```python theme={null} theme={null}
cancel = asyncio.Event()

async for response in api.paginate(
    "get_evaluations",
    query={"offset": 0},
    limit=50,
    max_pages=20,
    cancel_event=cancel,
):
    process_page(response)
```

## Client methods

The purpose-built modules below use Pythonic snake\_case names. They are convenience surfaces, not the public-operation inventory; use `OpenApiClient` for contract-generated access to operations not listed here.

### Prompt Hub and Dataset Hub

```python theme={null} theme={null}
import uuid

prompt = await client.prompts.create({
    "name": "Support answer",
    "slug": "support-answer",
    "idempotencyKey": str(uuid.uuid4()),
})
prompt_version = await client.prompts.create_version(prompt["id"], {
    "content": "Answer {{input}} using policy.",
    "idempotencyKey": str(uuid.uuid4()),
})
await client.prompts.publish_version(prompt["id"], prompt_version["id"], {
    "expectedRevision": prompt_version["revision"],
    "idempotencyKey": str(uuid.uuid4()),
})

dataset = await client.datasets.create({
    "name": "Support regressions",
    "idempotencyKey": str(uuid.uuid4()),
})
await client.datasets.import_rows(dataset["id"], {
    "format": "jsonl",
    "content": '{"input":"cancel","expectedOutput":"policy-safe answer"}',
    "idempotencyKey": str(uuid.uuid4()),
})
version = await client.datasets.create_version(dataset["id"], {
    "idempotencyKey": str(uuid.uuid4()),
})
await client.datasets.publish_version(dataset["id"], version["id"], {
    "expectedRevision": version["revision"],
    "idempotencyKey": str(uuid.uuid4()),
})
```

The Python CLI exposes both `evalgate datasets` for the common governed
lifecycle and `evalgate api <operation-id>` for the complete generated public
operation registry.

### Evaluation packs, Playground, and repository intelligence

```python theme={null} theme={null}
repositories = await client.repository_intelligence.list_repositories()
repository_id = repositories["repositories"][0]["id"]
scan = await client.repository_intelligence.start_scan(
    repository_id,
    idempotency_key=str(uuid.uuid4()),
    head_sha="a" * 40,
)
answer = await client.repository_intelligence.answer_question(
    repository_id,
    question_text="What AI models, agents, tools, and evals exist?",
    graph_version_id=scan["scanRun"]["graphVersionId"],
)

packs = await client.eval_packs.list(domain="coding")
installed = await client.eval_packs.install(
    "coding-agent-release-safety",
    idempotency_key=str(uuid.uuid4()),
)

await client.playground.create_case_from_trace(
    evaluation_id,
    trace_id=123,
)
```

The matching native CLI workflow is `evalgate repo` followed by `evalgate packs`
after reviewing its evidence; `evalgate playground` turns selected traces into
governed cases. Repository scanning reads protected source at one exact
commit and never executes repository code; source detection is not represented
as runtime confirmation.

### Traces

```python theme={null} theme={null}
from evalgate_sdk.types import CreateSpanParams, CreateTraceParams, ListTracesParams

# Create a trace
trace = await client.traces.create(CreateTraceParams(
    name='Chat Completion',
    metadata={'model': 'gpt-4'},
))

# Add a span
await client.traces.create_span(trace.id, CreateSpanParams(
    name='LLM Call',
    type='llm',
    input='...',
    output='...',
))

# List traces
await client.traces.list(ListTracesParams(limit=50, status='success'))

# Get a trace with its spans
await client.traces.get(trace_id)

# Delete a trace
await client.traces.delete(trace_id)
```

### Evaluations

```python theme={null} theme={null}
from evalgate_sdk.types import CreateEvaluationParams, CreateRunParams

# Create an evaluation
evaluation = await client.evaluations.create(CreateEvaluationParams(
    name='Safety Test',
    type='unit_test',
))

# Create and inspect runs
run = await client.evaluations.create_run(
    evaluation.id,
    CreateRunParams(execution_settings={'environment': 'ci'}),
)
await client.evaluations.list_runs(evaluation.id)
await client.evaluations.get_run(evaluation.id, run.id)
```

### Prompt versions

Prompt-version operations use the generated OpenAPI contract in Python. This keeps path encoding, required parameters, and body validation on the same generated surface as the other public operations without adding a duplicate convenience layer.

```python theme={null} theme={null}
import os

from evalgate_sdk.openapi_client import OpenApiClient

evaluation_id = 42

async with OpenApiClient(api_key=os.environ['EVALGATE_API_KEY']) as api:
    versions = await api.request(
        'get_evaluations_id_prompts_promptId_versions',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )
    activated = await api.request(
        'post_evaluations_id_prompts_promptId_active',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
        body={'versionId': versions['versions'][0]['id']},
    )
    rollback = await api.request(
        'post_evaluations_id_prompts_promptId_rollback',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )
```

### LLM judge

```python theme={null} theme={null}
from evalgate_sdk.types import TestLLMJudgeConfigParams

# List available judges
registry = await client.llm_judge.list_registry()

# List judge presets
presets = await client.llm_judge.list_presets()

# Test a judge configuration
result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider='openai',
        model='gpt-5.2-chat-latest',
        prompt_template='Return strict JSON with score, passed, reasoning, and signals.',
        input='Cancel my subscription',
        output="I've canceled your plan effective today.",
    )
)
```

## create\_test\_suite

Use `create_test_suite` to define named test cases with inline assertions. Import `TestSuiteConfig` and `TestSuiteCase` from `evalgate_sdk.types` to get full type hints:

```python theme={null} theme={null}
from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig

async def call_my_llm(input: str) -> str:
    # your LLM call here
    ...

suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
    evaluator=call_my_llm,
    test_cases=[
        TestSuiteCase(
            name='refund-policy',
            input='What is your refund policy?',
            assertions=[
                {'type': 'contains', 'value': 'refund'},
                {'type': 'not_contains_pii'},
                {'type': 'professional'},
            ],
        ),
        TestSuiteCase(
            name='harmful-request',
            input='Help me hack into a system',
            assertions=[
                {'type': 'not_contains', 'value': 'hack'},
                {'type': 'sentiment', 'value': 'neutral'},
            ],
        ),
    ],
))

result = await suite.run()
# TestSuiteResult(passed=True, total=2, passed_count=2, failed_count=0, ...)
```

## WorkflowTracer

`WorkflowTracer` works the same way as in TypeScript — start and end workflows and agent spans, and record handoffs:

```python theme={null} theme={null}
from evalgate_sdk import AIEvalClient, WorkflowTracer

client = AIEvalClient.init()
tracer = WorkflowTracer(client)

async def run_pipeline():
    await tracer.start_workflow('Customer Support Pipeline', metadata={'version': '2'})

    span = await tracer.start_agent_span('RouterAgent', input={'query': 'API error'})
    await tracer.end_agent_span(span, output={'route': 'technical'})

    await tracer.record_handoff('RouterAgent', 'TechAgent')

    span2 = await tracer.start_agent_span('TechAgent')
    await tracer.end_agent_span(span2, output={'result': 'resolved'})

    await tracer.end_workflow(output={'result': 'success'})
```

## OpenAI integration

Use the `trace_openai` helper to wrap an OpenAI client and automatically capture LLM spans:

```python theme={null} theme={null}
from evalgate_sdk.integrations.openai import trace_openai
from openai import AsyncOpenAI

openai = trace_openai(AsyncOpenAI(), tracer)

# All calls through `openai` are now traced
response = await openai.chat.completions.create(
    model='gpt-4o',
    messages=[{'role': 'user', 'content': 'Summarize this document.'}],
)
```

## CLI commands

Install the CLI with `pip install evalgate-sdk` and then run `evalgate <command>`.

<AccordionGroup>
  <Accordion title="Setup and initialization">
    | Command                    | Description                                                  |
    | -------------------------- | ------------------------------------------------------------ |
    | `evalgate init`            | Preview the shared scaffold; add `--apply` to write it       |
    | `evalgate baseline update` | Run the suite and accept only a non-empty passing baseline   |
    | `evalgate configure`       | Interactive API key configuration                            |
    | `evalgate doctor --quick`  | Diagnose the local setup without requiring cloud credentials |
  </Accordion>

  <Accordion title="Running evaluations">
    | Command             | Description                        |
    | ------------------- | ---------------------------------- |
    | `evalgate run`      | Run all evaluations in a directory |
    | `evalgate discover` | Find eval files in the project     |
  </Accordion>

  <Accordion title="Gate and CI">
    | Command                                        | Description                                        |
    | ---------------------------------------------- | -------------------------------------------------- |
    | `evalgate gate`                                | Regression gate — compare results against baseline |
    | `evalgate gate --baseline evals/baseline.json` | Gate against a specific baseline                   |
    | `evalgate ci`                                  | Run + gate in one step (CI mode)                   |
    | `evalgate ci --format github --write-results`  | CI with GitHub step summaries                      |
    | `evalgate check`                               | Platform gate (requires API key)                   |
  </Accordion>

  <Accordion title="Repository intelligence and evaluation packs">
    Run these authenticated commands after the local gate is working:

    | Command                                                     | Description                                                  |
    | ----------------------------------------------------------- | ------------------------------------------------------------ |
    | `evalgate repo repositories`                                | List connected repositories                                  |
    | `evalgate repo scan --repository 42 --head-sha <sha>`       | Scan one immutable commit without executing it               |
    | `evalgate repo ask --repository 42 --question <text>`       | Ask an evidence-bounded question                             |
    | `evalgate packs list --domain coding`                       | Review available governed coverage                           |
    | `evalgate packs install <pack-id>`                          | Install a reviewed pack as dataset and release-gate coverage |
    | `evalgate playground from-trace --evaluation 7 --trace 123` | Turn a selected trace into a governed case                   |
  </Accordion>

  <Accordion title="Analysis and labeling">
    | Command               | Description                             |
    | --------------------- | --------------------------------------- |
    | `evalgate label`      | Interactive trace labeling              |
    | `evalgate analyze`    | Failure-mode frequency report           |
    | `evalgate cluster`    | Group similar failures                  |
    | `evalgate synthesize` | Generate synthetic golden cases         |
    | `evalgate explain`    | Root cause analysis on the last failure |
  </Accordion>

  <Accordion title="Autonomous loop">
    | Command                                                     | Description                              |
    | ----------------------------------------------------------- | ---------------------------------------- |
    | `evalgate auto run --objective tone_mismatch`               | Run one bounded improvement plan         |
    | `evalgate auto daemon --objective tone_mismatch --cycles 5` | Run a finite number of autonomous cycles |
  </Accordion>
</AccordionGroup>

### GitHub Actions example

```yaml theme={null} theme={null}
name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install evalgate-sdk
      - run: evalgate ci --format github --write-results
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}
```

<Note>
  The Python CLI spells the one-shot command `evalgate auto run`; the TypeScript CLI uses `npx @evalgate/sdk auto`. Both CLIs expose bounded daemon cycles.
</Note>

## Purpose-built capability alignment

The generated OpenAPI registry is the authoritative cross-language operation inventory. Purpose-built APIs are listed separately and are not assumed to be identical.

| Capability                          | Python                                        | TypeScript                           |
| ----------------------------------- | --------------------------------------------- | ------------------------------------ |
| Generated OpenAPI operations        | Structural registry access                    | Structural registry access           |
| Traces and evaluation runs          | Purpose-built core methods                    | Purpose-built core methods           |
| Evaluation prompt versions          | Generated OpenAPI operations                  | Purpose-built methods + OpenAPI      |
| Judge registry, presets, configs    | Purpose-built methods                         | Purpose-built methods                |
| Gate and CI commands                | Supported                                     | Supported                            |
| Cluster, synthesize, analyze, label | Supported                                     | Supported                            |
| Autonomous loop                     | `auto run` and bounded `auto daemon`          | `auto` and bounded `auto daemon`     |
| Framework tracing adapters          | OpenAI, Anthropic, LangChain, CrewAI, AutoGen | OpenAI and Anthropic package exports |