Remote Runners

Remote Runners let EvalGate dispatch prompts, scorers, tools, workflows, and agent entrypoints into infrastructure that you control. The control plane retains job identity, policy, retry, and evidence authority; the worker receives one signed, immutable execution envelope at a time. Open Remote Runners at /remote-runners to register workers, inspect capacity and heartbeats, drain or revoke identities, monitor jobs, request cancellation, and review each job’s claims, state transitions, stream custody, terminal completion, and reconciliation decisions.

A remote job binds an exact release artifact ID, version ID, SHA-256 content hash, and canonical dev, staging, or prod environment. A runner cannot substitute another version, change the environment, or resolve “latest.”

Permissions

Action	Required scope
List runners and inspect jobs/evidence	`runs:read`
Enqueue or cancel remote jobs	`runs:write`
Register, drain, activate, rotate, or revoke a runner	`admin:org` and admin role
Claim, heartbeat, stream, or complete work	Short-lived runner credential with the matching `runner:*` scope

Runner endpoints do not accept a user session or ordinary API key as worker authority. The worker first exchanges its one-time bootstrap secret for a credential that expires within ten minutes and is bound to one organization and runner identity.

Security and identity model

Each registered runner has:

A stable organization-scoped runnerKey
A declared protocol version and content-hashed capability document
A maximum concurrent-job count enforced in the database
A one-time bootstrap token stored only as a SHA-256 hash
Short-lived, revocable worker credentials with least-privilege scopes
A derived HMAC-SHA256 job-signing key and visible key ID
Heartbeat, capacity, last-contact, drain, loss, and revocation state

Rotating a bootstrap token revokes all outstanding short-lived credentials. Revocation is permanent for that identity; register a replacement runner rather than reactivating a revoked credential boundary.

The bootstrap token is displayed once. Save it directly into your secret manager before closing the dialog. EvalGate cannot recover the plaintext value.

Register a runner

Open /remote-runners and choose Register runner.
Enter a display name and stable runner key.
Declare immutable runtime identifiers, such as nodejs20 or python3.12.
Select the supported execution kinds.
Set the maximum concurrent-job capacity.
Review the default-deny isolation boundary and choose Register identity.
Copy the displayed environment variables and worker start command into your deployment secret and workload configuration.

The current registration flow declares network deny_all, streaming and cancellation support, one-hour CPU, 64 GiB memory, 24-hour wall time, and a 100 MiB output ceiling. The API accepts a narrower capability document when the worker has smaller limits or a reviewed HTTPS origin allowlist.

Capability negotiation

Before a claim is created, the control plane compares every job requirement with the runner declaration:

Requirement	Enforcement
Execution kind	Must appear in the runner’s supported kinds
Runtime	Exact immutable runtime identifier match
CPU, memory, wall time, and output	Every requested limit must be at or below the runner ceiling
Network	`deny_all`, or every required HTTPS origin must appear in the runner allowlist
Streaming and cancellation	Required support must be explicitly declared
Concurrency	Active claims must remain below the registered maximum

An incompatible runner simply does not receive the job. EvalGate does not silently relax a resource or network requirement to make a claim succeed.

Start a TypeScript worker

The TypeScript SDK verifies the organization binding, key ID, validity window, canonical payload hash, HMAC signature, claim identity, and nonce before returning an assignment to application code.

import {
  RemoteRunnerWorker,
  createRemoteRunnerCompletion,
  createRemoteRunnerStreamChunk,
} from "@evalgate/sdk";

const worker = new RemoteRunnerWorker({
  baseUrl: process.env.EVALGATE_BASE_URL!,
  runnerId: process.env.EVALGATE_RUNNER_ID!,
  bootstrapToken: process.env.EVALGATE_RUNNER_BOOTSTRAP_TOKEN!,
});

const assignment = await worker.claim();
if (!assignment) process.exit(0);

const started = createRemoteRunnerStreamChunk({
  jobId: assignment.signedJob.envelope.jobId,
  claimId: assignment.claim.id,
  dispatchAttempt: assignment.claim.dispatchAttempt,
  sequence: 0,
  kind: "log",
  previousChecksum: null,
  payload: { level: "info", message: "execution started" },
});
await worker.stream(started);

await worker.heartbeat(assignment.claim.id, 60);

await worker.complete(createRemoteRunnerCompletion({
  jobId: assignment.signedJob.envelope.jobId,
  claimId: assignment.claim.id,
  runnerId: process.env.EVALGATE_RUNNER_ID!,
  organizationId: assignment.signedJob.envelope.organizationId,
  dispatchAttempt: assignment.claim.dispatchAttempt,
  status: "succeeded",
  inputHash: assignment.signedJob.envelope.inputHash,
  resultHash: resultHash,
  finalStreamChecksum: started.checksum,
  evidence: {
    evaluationRunId,
    testResultIds,
    costRecordIds,
    modelCallIds,
    traceId,
  },
  completedAt: new Date().toISOString(),
}));

Start a Python worker

The Python SDK implements the same canonical hashing, signing verification, replay protection, credential refresh, and endpoint contract.

import os
from evalgate_sdk import (
    RemoteRunnerWorker,
    create_remote_runner_completion,
    create_remote_runner_stream_chunk,
)

async with RemoteRunnerWorker(
    base_url=os.environ["EVALGATE_BASE_URL"],
    runner_id=os.environ["EVALGATE_RUNNER_ID"],
    bootstrap_token=os.environ["EVALGATE_RUNNER_BOOTSTRAP_TOKEN"],
) as worker:
    assignment = await worker.claim()
    if assignment is None:
        return

    chunk = create_remote_runner_stream_chunk(
        job_id=assignment["signedJob"]["envelope"]["jobId"],
        claim_id=assignment["claim"]["id"],
        dispatch_attempt=assignment["claim"]["dispatchAttempt"],
        sequence=0,
        kind="trajectory",
        previous_checksum=None,
        payload={"event": "agent_started"},
    )
    await worker.stream(chunk)
    await worker.heartbeat(assignment["claim"]["id"])
    await worker.complete(create_remote_runner_completion(
        jobId=assignment["signedJob"]["envelope"]["jobId"],
        claimId=assignment["claim"]["id"],
        runnerId=os.environ["EVALGATE_RUNNER_ID"],
        organizationId=assignment["signedJob"]["envelope"]["organizationId"],
        dispatchAttempt=assignment["claim"]["dispatchAttempt"],
        status="succeeded",
        inputHash=assignment["signedJob"]["envelope"]["inputHash"],
        resultHash=result_hash,
        finalStreamChecksum=chunk["checksum"],
        evidence=evidence,
        completedAt=completed_at,
    ))

Enqueue an immutable job

Use POST /api/remote-runners/jobs with a runs:write API key or authenticated session. The request contains:

A stable idempotency key
Subject kind, artifact ID, exact version ID, and dev, staging, or prod environment
A JSON input snapshot
Execution kind, runtime, resource limits, network policy, and streaming/cancellation requirements

The server loads the exact release version in the caller’s organization, computes and persists its content hash when needed, checks the version’s environment, and creates the queued job and audit event atomically. Repeating the same idempotency key with identical intent returns the original job; different intent returns a conflict.

Signed claim lifecycle

queued → claimed → running → succeeded | failed
   │         │          │
   └─────────┴──────────┴→ cancel_requested → cancelled
             │
             └→ lost → queued (bounded retry) | cancelled

Claim insertion is serialized at the database boundary. A job can have only one active claim and one claim per dispatch attempt. The signed envelope includes the organization, job, attempt, idempotency key, exact subject identity, input hash, requirements, issue/expiry timestamps, and a single-use nonce. The first heartbeat transitions a claimed job to running and extends the lease for 15–120 seconds. It also returns cancelRequested and the human cancellation rationale so the worker can stop at a safe boundary.

Stream logs and trajectories

Each log or trajectory chunk includes:

Job, claim, and dispatch-attempt identity
A zero-based contiguous sequence
The prior chunk checksum, or null for sequence zero
A deterministic payload hash
A checksum over identity, sequence, kind, prior checksum, payload hash, and emission time

The service and database both reject gaps, reordered chunks, changed claim identity, duplicate sequence numbers, duplicate checksums, and appends to an expired claim. The terminal completion must close the exact last stored checksum.

Complete with normal EvalGate evidence

A completion is accepted only for the active claim and attempt. It binds the input hash, result hash, terminal status, final stream checksum, and normal EvalGate evidence:

Evaluation run ID
Test-result IDs
Cost-record IDs
Model-call ledger IDs
Trace ID

All referenced evidence must belong to the same organization and the exact evaluation run named by the completion; each evidence row may be attached to only one accepted remote completion. Completion insertion, evidence joins, job finalization, claim release, runner-capacity decrement, and the terminal event occur in one transaction. Exact completion replay returns duplicate. A different completion after acceptance returns conflict. A completion from an earlier dispatch attempt returns stale. These decisions are stored with their checksum, rationale, and reported costs rather than discarded.

Cancellation and runner loss

Queued and lost jobs cancel immediately. Claimed and running jobs enter cancel_requested; the worker sees that state on heartbeat. If cancellation wins the terminal race, the job becomes cancelled while already-incurred provider or sandbox costs remain authoritative. When a heartbeat lease expires, the reaper:

Marks the claim and runner lost.
Releases the runner’s active capacity.
Appends a durable loss event.
Requeues the job with no active claim when fewer than three attempts have been dispatched.
Uses a new nonce and incremented attempt for the next compatible runner.

EvalGate invokes bounded recovery from the existing CRON_SECRET-authenticated job runner. Organization admins can also trigger the same bounded recovery from the control plane during incident response; both paths retain the durable loss and requeue events. An older worker cannot complete the replacement attempt. Accepted results, test results, model calls, and cost records have unique custody constraints, preventing duplicate results or charges.

Operate and troubleshoot

Symptom	What to check
Session returns `401`	Runner ID and current bootstrap token; rotate if the token may be exposed
Session returns `503`	`REMOTE_RUNNER_SIGNING_SECRET` must be configured with at least 32 characters
Claim returns `204`	No compatible queued job, or another runner claimed the available work
Claim returns `503`	Runner is draining, lost, revoked, or at concurrent capacity
Job stays claimed	Worker must heartbeat; the first heartbeat records execution start
Stream returns `422` or `409`	Verify sequence, prior checksum, claim/attempt identity, and active lease
Completion is `stale`	A newer dispatch attempt replaced this runner after lease loss
Completion is `conflict`	Another terminal checksum was already accepted for this job

Use the Jobs tab and Evidence action to inspect the complete state timeline and reconciliation history before retrying or rotating a worker.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Remote Runners

Remote Runners

Permissions

Security and identity model

Register a runner

Capability negotiation

Start a TypeScript worker

Start a Python worker

Enqueue an immutable job

Signed claim lifecycle

Stream logs and trajectories

Complete with normal EvalGate evidence

Cancellation and runner loss

Operate and troubleshoot

​Remote Runners

​Permissions

​Security and identity model

​Register a runner

​Capability negotiation

​Start a TypeScript worker

​Start a Python worker

​Enqueue an immutable job

​Signed claim lifecycle

​Stream logs and trajectories

​Complete with normal EvalGate evidence

​Cancellation and runner loss

​Operate and troubleshoot

Remote Runners

Permissions

Security and identity model

Register a runner

Capability negotiation

Start a TypeScript worker

Start a Python worker

Enqueue an immutable job

Signed claim lifecycle

Stream logs and trajectories

Complete with normal EvalGate evidence

Cancellation and runner loss

Operate and troubleshoot