> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Remote Runners

> Execute immutable EvalGate release artifacts in customer-controlled infrastructure with signed jobs, bounded capabilities, and complete evidence custody.

# Remote Runners

Remote Runners let EvalGate dispatch prompts, scorers, tools, workflows, and agent entrypoints into infrastructure that you control. The control plane retains job identity, policy, retry, and evidence authority; the worker receives one signed, immutable execution envelope at a time.

Open **Remote Runners** at `/remote-runners` to register workers, inspect capacity and heartbeats, drain or revoke identities, monitor jobs, request cancellation, and review each job's claims, state transitions, stream custody, terminal completion, and reconciliation decisions.

<Info>
  A remote job binds an exact release artifact ID, version ID, SHA-256 content hash, and canonical `dev`, `staging`, or `prod` environment. A runner cannot substitute another version, change the environment, or resolve “latest.”
</Info>

## Permissions

| Action                                                | Required scope                                                   |
| ----------------------------------------------------- | ---------------------------------------------------------------- |
| List runners and inspect jobs/evidence                | `runs:read`                                                      |
| Enqueue or cancel remote jobs                         | `runs:write`                                                     |
| Register, drain, activate, rotate, or revoke a runner | `admin:org` and admin role                                       |
| Claim, heartbeat, stream, or complete work            | Short-lived runner credential with the matching `runner:*` scope |

Runner endpoints do not accept a user session or ordinary API key as worker authority. The worker first exchanges its one-time bootstrap secret for a credential that expires within ten minutes and is bound to one organization and runner identity.

## Security and identity model

Each registered runner has:

* A stable organization-scoped `runnerKey`
* A declared protocol version and content-hashed capability document
* A maximum concurrent-job count enforced in the database
* A one-time bootstrap token stored only as a SHA-256 hash
* Short-lived, revocable worker credentials with least-privilege scopes
* A derived HMAC-SHA256 job-signing key and visible key ID
* Heartbeat, capacity, last-contact, drain, loss, and revocation state

Rotating a bootstrap token revokes all outstanding short-lived credentials. Revocation is permanent for that identity; register a replacement runner rather than reactivating a revoked credential boundary.

<Warning>
  The bootstrap token is displayed once. Save it directly into your secret manager before closing the dialog. EvalGate cannot recover the plaintext value.
</Warning>

## Register a runner

1. Open `/remote-runners` and choose **Register runner**.
2. Enter a display name and stable runner key.
3. Declare immutable runtime identifiers, such as `nodejs20` or `python3.12`.
4. Select the supported execution kinds.
5. Set the maximum concurrent-job capacity.
6. Review the default-deny isolation boundary and choose **Register identity**.
7. Copy the displayed environment variables and worker start command into your deployment secret and workload configuration.

The current registration flow declares network `deny_all`, streaming and cancellation support, one-hour CPU, 64 GiB memory, 24-hour wall time, and a 100 MiB output ceiling. The API accepts a narrower capability document when the worker has smaller limits or a reviewed HTTPS origin allowlist.

## Capability negotiation

Before a claim is created, the control plane compares every job requirement with the runner declaration:

| Requirement                        | Enforcement                                                                    |
| ---------------------------------- | ------------------------------------------------------------------------------ |
| Execution kind                     | Must appear in the runner's supported kinds                                    |
| Runtime                            | Exact immutable runtime identifier match                                       |
| CPU, memory, wall time, and output | Every requested limit must be at or below the runner ceiling                   |
| Network                            | `deny_all`, or every required HTTPS origin must appear in the runner allowlist |
| Streaming and cancellation         | Required support must be explicitly declared                                   |
| Concurrency                        | Active claims must remain below the registered maximum                         |

An incompatible runner simply does not receive the job. EvalGate does not silently relax a resource or network requirement to make a claim succeed.

## Start a TypeScript worker

The TypeScript SDK verifies the organization binding, key ID, validity window, canonical payload hash, HMAC signature, claim identity, and nonce before returning an assignment to application code.

```ts theme={null}
import {
  RemoteRunnerWorker,
  createRemoteRunnerCompletion,
  createRemoteRunnerStreamChunk,
} from "@evalgate/sdk";

const worker = new RemoteRunnerWorker({
  baseUrl: process.env.EVALGATE_BASE_URL!,
  runnerId: process.env.EVALGATE_RUNNER_ID!,
  bootstrapToken: process.env.EVALGATE_RUNNER_BOOTSTRAP_TOKEN!,
});

const assignment = await worker.claim();
if (!assignment) process.exit(0);

const started = createRemoteRunnerStreamChunk({
  jobId: assignment.signedJob.envelope.jobId,
  claimId: assignment.claim.id,
  dispatchAttempt: assignment.claim.dispatchAttempt,
  sequence: 0,
  kind: "log",
  previousChecksum: null,
  payload: { level: "info", message: "execution started" },
});
await worker.stream(started);

await worker.heartbeat(assignment.claim.id, 60);

await worker.complete(createRemoteRunnerCompletion({
  jobId: assignment.signedJob.envelope.jobId,
  claimId: assignment.claim.id,
  runnerId: process.env.EVALGATE_RUNNER_ID!,
  organizationId: assignment.signedJob.envelope.organizationId,
  dispatchAttempt: assignment.claim.dispatchAttempt,
  status: "succeeded",
  inputHash: assignment.signedJob.envelope.inputHash,
  resultHash: resultHash,
  finalStreamChecksum: started.checksum,
  evidence: {
    evaluationRunId,
    testResultIds,
    costRecordIds,
    modelCallIds,
    traceId,
  },
  completedAt: new Date().toISOString(),
}));
```

## Start a Python worker

The Python SDK implements the same canonical hashing, signing verification, replay protection, credential refresh, and endpoint contract.

```python theme={null}
import os
from evalgate_sdk import (
    RemoteRunnerWorker,
    create_remote_runner_completion,
    create_remote_runner_stream_chunk,
)

async with RemoteRunnerWorker(
    base_url=os.environ["EVALGATE_BASE_URL"],
    runner_id=os.environ["EVALGATE_RUNNER_ID"],
    bootstrap_token=os.environ["EVALGATE_RUNNER_BOOTSTRAP_TOKEN"],
) as worker:
    assignment = await worker.claim()
    if assignment is None:
        return

    chunk = create_remote_runner_stream_chunk(
        job_id=assignment["signedJob"]["envelope"]["jobId"],
        claim_id=assignment["claim"]["id"],
        dispatch_attempt=assignment["claim"]["dispatchAttempt"],
        sequence=0,
        kind="trajectory",
        previous_checksum=None,
        payload={"event": "agent_started"},
    )
    await worker.stream(chunk)
    await worker.heartbeat(assignment["claim"]["id"])
    await worker.complete(create_remote_runner_completion(
        jobId=assignment["signedJob"]["envelope"]["jobId"],
        claimId=assignment["claim"]["id"],
        runnerId=os.environ["EVALGATE_RUNNER_ID"],
        organizationId=assignment["signedJob"]["envelope"]["organizationId"],
        dispatchAttempt=assignment["claim"]["dispatchAttempt"],
        status="succeeded",
        inputHash=assignment["signedJob"]["envelope"]["inputHash"],
        resultHash=result_hash,
        finalStreamChecksum=chunk["checksum"],
        evidence=evidence,
        completedAt=completed_at,
    ))
```

## Enqueue an immutable job

Use `POST /api/remote-runners/jobs` with a `runs:write` API key or authenticated session. The request contains:

* A stable idempotency key
* Subject kind, artifact ID, exact version ID, and `dev`, `staging`, or `prod` environment
* A JSON input snapshot
* Execution kind, runtime, resource limits, network policy, and streaming/cancellation requirements

The server loads the exact release version in the caller's organization, computes and persists its content hash when needed, checks the version's environment, and creates the queued job and audit event atomically. Repeating the same idempotency key with identical intent returns the original job; different intent returns a conflict.

## Signed claim lifecycle

```text theme={null}
queued → claimed → running → succeeded | failed
   │         │          │
   └─────────┴──────────┴→ cancel_requested → cancelled
             │
             └→ lost → queued (bounded retry) | cancelled
```

Claim insertion is serialized at the database boundary. A job can have only one active claim and one claim per dispatch attempt. The signed envelope includes the organization, job, attempt, idempotency key, exact subject identity, input hash, requirements, issue/expiry timestamps, and a single-use nonce.

The first heartbeat transitions a claimed job to running and extends the lease for 15–120 seconds. It also returns `cancelRequested` and the human cancellation rationale so the worker can stop at a safe boundary.

## Stream logs and trajectories

Each `log` or `trajectory` chunk includes:

* Job, claim, and dispatch-attempt identity
* A zero-based contiguous sequence
* The prior chunk checksum, or `null` for sequence zero
* A deterministic payload hash
* A checksum over identity, sequence, kind, prior checksum, payload hash, and emission time

The service and database both reject gaps, reordered chunks, changed claim identity, duplicate sequence numbers, duplicate checksums, and appends to an expired claim. The terminal completion must close the exact last stored checksum.

## Complete with normal EvalGate evidence

A completion is accepted only for the active claim and attempt. It binds the input hash, result hash, terminal status, final stream checksum, and normal EvalGate evidence:

* Evaluation run ID
* Test-result IDs
* Cost-record IDs
* Model-call ledger IDs
* Trace ID

All referenced evidence must belong to the same organization and the exact evaluation run named by the completion; each evidence row may be attached to only one accepted remote completion. Completion insertion, evidence joins, job finalization, claim release, runner-capacity decrement, and the terminal event occur in one transaction.

Exact completion replay returns `duplicate`. A different completion after acceptance returns `conflict`. A completion from an earlier dispatch attempt returns `stale`. These decisions are stored with their checksum, rationale, and reported costs rather than discarded.

## Cancellation and runner loss

Queued and lost jobs cancel immediately. Claimed and running jobs enter `cancel_requested`; the worker sees that state on heartbeat. If cancellation wins the terminal race, the job becomes cancelled while already-incurred provider or sandbox costs remain authoritative.

When a heartbeat lease expires, the reaper:

1. Marks the claim and runner lost.
2. Releases the runner's active capacity.
3. Appends a durable loss event.
4. Requeues the job with no active claim when fewer than three attempts have been dispatched.
5. Uses a new nonce and incremented attempt for the next compatible runner.

EvalGate invokes bounded recovery from the existing `CRON_SECRET`-authenticated job runner. Organization admins can also trigger the same bounded recovery from the control plane during incident response; both paths retain the durable loss and requeue events.

An older worker cannot complete the replacement attempt. Accepted results, test results, model calls, and cost records have unique custody constraints, preventing duplicate results or charges.

## Operate and troubleshoot

| Symptom                       | What to check                                                                 |
| ----------------------------- | ----------------------------------------------------------------------------- |
| Session returns `401`         | Runner ID and current bootstrap token; rotate if the token may be exposed     |
| Session returns `503`         | `REMOTE_RUNNER_SIGNING_SECRET` must be configured with at least 32 characters |
| Claim returns `204`           | No compatible queued job, or another runner claimed the available work        |
| Claim returns `503`           | Runner is draining, lost, revoked, or at concurrent capacity                  |
| Job stays claimed             | Worker must heartbeat; the first heartbeat records execution start            |
| Stream returns `422` or `409` | Verify sequence, prior checksum, claim/attempt identity, and active lease     |
| Completion is `stale`         | A newer dispatch attempt replaced this runner after lease loss                |
| Completion is `conflict`      | Another terminal checksum was already accepted for this job                   |

Use the **Jobs** tab and **Evidence** action to inspect the complete state timeline and reconciliation history before retrying or rotating a worker.