# EvalGate — complete agent integration reference

> Current application and SDK source contract: 3.7.9. EvalGate is a controlled
> beta. Use the published OpenAPI document and feature-status inventory as the
> authorities for operations and availability.

## Product boundary

EvalGate is a managed evaluation control plane for AI applications and agent
workflows. It connects:

- exact-repository and commit context;
- traces, spans, tool calls, trajectories, costs, and outcomes;
- deterministic, programmatic, human, and model-judge evaluation;
- reviewed golden cases and baseline lineage;
- local and CI regression gates;
- continuous evaluation and release evidence; and
- governed runtime decisions for integrated calls.

EvalGate is not a model provider. Customers bring provider credentials for
model-backed workflows and receive provider charges separately. EvalGate is not
an identity provider, universal proxy, payment processor, or production sandbox.

## Discovery

| Purpose | URL |
| --- | --- |
| Short agent guide | `https://www.evalgate.com/llms.txt` |
| OpenAPI 3.1 | `https://www.evalgate.com/openapi.json` |
| API catalog | `https://www.evalgate.com/.well-known/api-catalog` |
| AI resource catalog | `https://www.evalgate.com/.well-known/ai-catalog.json` |
| Authentication walkthrough | `https://www.evalgate.com/auth.md` |
| Evidence-backed capability map | `https://www.evalgate.com/capabilities.md` |
| RFC 9728 API metadata | `https://www.evalgate.com/.well-known/oauth-protected-resource` |
| Agent Skill index | `https://www.evalgate.com/.well-known/agent-skills/index.json` |
| A2A v1 card | `https://www.evalgate.com/.well-known/agent-card.json` |
| Product MCP card | `https://www.evalgate.com/.well-known/mcp/server-card.json` |
| Documentation MCP card | `https://www.evalgate.com/.well-known/mcp/docs-server-card.json` |
| Authenticated NLWeb retrieval | `https://www.evalgate.com/ask` |
| API versioning policy | `https://www.evalgate.com/docs/api/versioning` |
| Pricing | `https://www.evalgate.com/pricing.md` |
| Feature status | `https://www.evalgate.com/docs/platform/feature-status` |

## Identity and credentials

The browser account flow supports interactive GitHub and Google sign-in. After
sign-in, a person creates or joins an organization, connects a repository, and
creates an organization-scoped API key. Hosted SDK, CLI, REST, MCP, A2A, and
retrieval calls use:

```http
Authorization: Bearer <EVALGATE_API_KEY>
```

Use `EVALGATE_ORGANIZATION_ID` when an SDK workflow requires an explicit
organization identifier. Direct REST routes generally derive tenant context from
the key. Never accept an organization ID from untrusted input as authorization.

EvalGate publishes RFC 9728 protected-resource metadata and scope names for its
actual bearer-key contract. It does not currently publish an RFC 8414 OAuth
authorization server, token endpoint, or dynamic client-registration endpoint.
Do not fabricate OAuth URLs.

For a machine that cannot complete browser sign-in, call
`POST /api/agent-setup/start`, show the returned verification URI and user code
to a person, and poll `POST /api/agent-setup/poll` at the advertised interval.
The person signs in with GitHub or Google, selects the accountable organization,
and reviews the requested non-administrative scopes. Approval returns one
90-day organization API key exactly once. This handoff does not create an
identity anonymously and is not an OAuth authorization-server claim.

## Existing capability matrix

| Capability | Available evidence | Do not overstate |
| --- | --- | --- |
| Proactive synthetic and adversarial testing | Red-team risk packs and campaigns, `evalgate synthesize`, durable synthesis runs, and quarantined candidate review generate dimension, paraphrase, harder-variant, and adversarial coverage before a production incident | No exhaustive arbitrary tool-schema fuzzing, universal API-timeout simulation, or production-like sandbox claim |
| Trajectory reconstruction and comparison | `/trajectory-analysis` reconstructs ordered participants, handoffs, tools, model calls, costs, evidence links, integrity gaps, scores, reports, and baseline/candidate metrics; SDK matchers enforce sequence and tool-argument policies | The web UI is not yet a complete node-by-node side-by-side graph-diff renderer |
| Human-aligned judge calibration | `/calibration` versions human anchors and mappings and tracks agreement, correlation, TPR/TNR, bootstrap intervals, parse failures, drift, diffs, and release gates; `/llm-judge` reviews disagreement-linked rubric, threshold, and weighting proposals | Cohen's kappa is not the primary published statistic |
| Tiered execution and cost control | Deterministic local/offline gates can run on pull requests while teams reserve hosted judges, pairwise experiments, and broad suites for merge queues or staging; cost evidence, budgets, BYOK, and `evalgate check` bound heavier work | CI event selection remains the team's workflow policy, not a universal scheduler |
| CI/CD delivery | The CLI generates a GitHub Actions workflow and emits annotations/step summaries; the public CI guide includes a GitLab CI recipe using the same portable gate | No dedicated CircleCI orb or native plugin is published |
| Alerts and webhooks | Signed organization webhooks deliver cataloged evaluation, trace, alert, dashboard, mutation, and proposal events; Slack supports an optional configured App plus Incoming Webhook fallback | No native Microsoft Teams, Jira, or PagerDuty delivery is advertised |
| OpenTelemetry and OpenInference | Authenticated OTLP/HTTP ingestion preserves trace, span, tool, model, token, latency, error, and compatible OpenInference evidence; the TypeScript SDK exports OTLP | This is trace integration, not a full replacement for a distributed-observability backend |
| Prompt and experiment lifecycle | Prompt Hub stores immutable content-hashed versions with review, environment bindings, compare, usage attribution, publish, and rollback; experiments freeze baseline/candidate prompt, model, tool, scorer, routing, and dataset variants | Reproducible comparison does not imply randomized assignment for every production request |
| Governed shadow comparison | Deployable Assets can execute a 100% non-serving candidate for requests routed through the governed invocation endpoint; failures are evidence and never replace a valid serving response | No transparent mirroring of arbitrary traffic that bypasses EvalGate |
| Export and SDK breadth | Bounded JSON/CSV/JSONL exports plus official TypeScript and Python SDKs | No provider-specific fine-tuning mapper and no official Go, Java, Rust, or C# SDK today |

See `https://www.evalgate.com/capabilities.md` for exact entry points and current
boundaries.

## Least-privilege agent scopes

| Surface | Required scope |
| --- | --- |
| Documentation MCP | `docs:read` |
| A2A evaluation and run inspection | `agent:read` |
| A2A bounded evaluation start | `agent:read`, `agent:execute` |
| Product MCP | Tool-specific; card declares `eval:read`, `runs:read`, `admin:org` |

A2A execution starts an existing evaluation in a bounded environment. It does
not create or mutate repositories, credentials, billing, prompts, or evaluation
definitions. Product MCP tool availability is filtered to the key's scopes.

## Installation

```bash
npm install @evalgate/sdk
# or
pip install evalgate-sdk
```

First complete signed-in repository setup at `https://www.evalgate.com/setup`.
The setup flow anchors the GitHub installation, organization, repository,
branch, repository root, and install preset, then displays a least-privilege key
once.

```bash
npx evalgate auth configure --api-key <install-key>
npx @evalgate/sdk init
npx @evalgate/sdk init --apply
npx @evalgate/sdk baseline update
npx @evalgate/sdk gate --format github
```

The deterministic gate may run without a network request after installation,
but it is not a separate anonymous onboarding mode. Dashboard evidence, CI, and
hosted workflows remain anchored to the account, repository, and scoped key.

## TypeScript client

```ts
import { AIEvalClient } from "@evalgate/sdk";

const client = AIEvalClient.init();
// Reads EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID.
```

Use the typed client and exported helpers for traces, evaluations, assertions,
judge workflows, baseline/gate artifacts, and repository-aware workflows. Refer
to `https://www.evalgate.com/docs/sdk/typescript` for current examples.

## Python client

```python
from evalgate_sdk import AIEvalClient

client = AIEvalClient.init()
```

The Python package includes the CLI and typed client workflows. Refer to
`https://www.evalgate.com/docs/sdk/python` for current async and sync examples.

## Framework integrations

| Framework | Verified boundary | Notes |
| --- | --- | --- |
| CrewAI | `traceCrewAI` from `@evalgate/sdk` | Wraps a structural `kickoff` method; framework installed separately |
| AutoGen | `traceAutoGen` from `@evalgate/sdk` | Wraps a structural `initiate_chat` method; framework installed separately |
| Vercel AI SDK | `traceVercelAIModel` from `@evalgate/sdk/integrations/vercel-ai` | Wraps Language Model v3-compatible generation and streaming |
| LangChain | SDK tracing wrapper and offline recipe | Preserve chain and agent evidence needed by the gate |
| LiteLLM | OpenAI-compatible Model Gateway backend | Gateway routing path, not a dedicated tracing wrapper |
| LlamaIndex | OpenTelemetry/OpenInference export | No first-class EvalGate wrapper is claimed |
| Generic Python / DSPy | Python SDK or OpenTelemetry/OpenInference export | No dedicated DSPy adapter is claimed |
| MCP clients | Streamable HTTP product and docs servers | Organization API key required |

The adapters collect evidence; they do not replace framework configuration,
provider credentials, network policy, or provider-side spend limits.

## A2A v1

Transport: `POST https://www.evalgate.com/api/a2a`

Protocol binding: JSON-RPC, A2A version 1.0. The public card describes input and
output media types, security schemes, scopes, and available skills. Current
skills cover evaluation inspection, run status, and separately scoped bounded
evaluation execution.

Clients should send the version header declared by the transport contract and
handle A2A task-not-found and under-scoped errors as typed failures.

## MCP

Product transport:

```text
POST https://www.evalgate.com/api/mcp
```

Documentation transport:

```text
POST https://www.evalgate.com/api/mcp/docs
```

Both use Streamable HTTP and bearer API keys. The documentation server is
read-only and exposes `docs.list`, `docs.search`, and `docs.get` over a bounded
catalog of published EvalGate documentation. The product server exposes
organization-scoped quality-evidence and planning tools described by its public
server card.

Do not send standards-based MCP traffic to legacy proprietary tool routes.

## REST API

Use `https://www.evalgate.com/openapi.json` instead of copying an endpoint list
from this file. The OpenAPI document is the operation, parameter, request,
response, error, authentication, and deprecation authority.

Rules for machine clients:

1. Use the unique `operationId` and typed schema.
2. Send an organization-scoped bearer key on protected operations.
3. Provide an idempotency key when the operation declares one.
4. Follow pagination bounds instead of assuming an unbounded list.
5. Treat `401`, `403`, `404`, `409`, `422`, `429`, and `5xx` as distinct states.
6. Honor published rate-limit and `Retry-After` headers.
7. Follow OpenAPI deprecation markers, the RFC 9745 `Deprecation` header, and
   any linked migration guide before a published RFC 8594 `Sunset` date.
8. Never interpret a blocked, incomplete, provider-failed, or cancelled run as a pass.

Authenticated `GET` or `POST https://www.evalgate.com/ask` provides bounded,
read-only retrieval over reviewed public EvalGate documentation. It requires
`docs:read`, supports JSON or Server-Sent Events, and does not expose private
repository or customer data.

## Evaluation evidence model

A useful release packet preserves:

- evaluation and case identity;
- exact baseline, source commit, and configuration hash;
- deterministic and judge result details;
- model/provider/rubric/parser provenance;
- trace, tool, trajectory, cost, and side-effect evidence;
- reviewer, approval, and promotion events;
- denied, blocked, cancelled, timed-out, and incomplete states; and
- the report or CI artifact attached to the release decision.

The conceptual loop is:

```text
repository -> trace -> cluster -> synthesize -> gate -> review -> improve -> ship
```

Generated or clustered cases are candidates until reviewed. Do not promote model
output directly into a trusted baseline without provenance and approval.

## Runtime policy boundary

Integrated EvalGate paths can evaluate:

- gateway active state and allowed use case;
- provider and model allowlists/blocklists;
- declared data and zero-data-retention policy;
- judge calibration requirements;
- request and organization budgets;
- data classification and egress destination;
- organization context and request correlation;
- runtime, connector, action, side-effect, risk, and manifest policy; and
- required human approval for protected actions.

Decisions preserve a reason code and policy snapshot or evidence hash. The
control applies only when the call is routed through the relevant EvalGate
integration. Direct provider calls outside that boundary are not intercepted.

## Pricing

| Plan | Monthly price | Included persisted results |
| --- | ---: | ---: |
| Developer | Free | 10,000 |
| Team | $99 | 100,000 |
| Scale | $499 | 1,000,000 |
| Business | $1,500 | 5,000,000 |

All self-serve plans continue uncapped at $1 per 1,000 additional persisted
results. Provider inference remains BYOK and is billed separately. Strategic
Enterprise provides a reviewed commercial path for more than 5 million monthly
results, multiple organizations, procurement/invoicing, negotiated volume
schedules, and contract-specific rollout terms.

## Production and assurance boundary

EvalGate is a controlled beta. Check the maintained feature-status inventory
before adopting any critical surface. Beta and Experimental labels are not
silently upgraded by marketing copy.

EvalGate does not publish a default uptime SLA, SOC 2 certification, universal
data-residency commitment, or guaranteed support-response time today.
Compliance-named governance templates are product configuration presets, not
external certifications. Strategic Enterprise terms exist only when written in
a signed agreement.

The hosted service is managed. Public SDK packages, examples, skills, OpenAPI,
MCP/A2A cards, and other protocol contracts are intended integration surfaces;
they do not advertise the entire hosted platform as self-hostable or open source.

## Public proof policy

Do not infer customer adoption from a synthetic tutorial. Named logos,
testimonials, production metrics, case studies, independent reviews, and
certifications are valid proof only when the public source identifies and
supports them. EvalGate's support-chatbot article is explicitly an illustrative
playbook, not a customer outcome.

## Current references

- Setup: https://www.evalgate.com/setup
- Docs: https://www.evalgate.com/docs
- Agent framework guide: https://www.evalgate.com/docs/guides/agent-framework-integrations
- Runtime controls: https://www.evalgate.com/docs/platform/runtime-controls
- Production adoption: https://www.evalgate.com/docs/platform/production-adoption
- Authentication: https://www.evalgate.com/auth.md
- OpenAPI: https://www.evalgate.com/openapi.json
- Pricing: https://www.evalgate.com/pricing.md
- npm: https://www.npmjs.com/package/@evalgate/sdk
- PyPI: https://pypi.org/project/evalgate-sdk/