Skip to main content
Does a deeper agent run find more of what matters, and what does the extra depth cost? This workflow answers that for one process in one codebase: run the agent at a few depths, score each run against an answer key a person wrote, and lay quality against cost side by side.
This workflow is Experimental. It ships in the TypeScript 4.0.0 source candidate of @evalgate/sdk, which is not published yet, and it is TypeScript-only until the Python port. Claude Code is supported, and Claude Managed Agents, OpenAI Agents API and Gemini Managed Agents imports are Experimental. Every number is self-reported by the agent’s own telemetry and is not verified against gateway or billing data. No hosted view exists yet.

​
What EvalGate does and does not do

Everything below runs locally through the CLI. Import, scoring, measures, views and the PDF never open a network connection, never call a model and never read a credential. The coding agent itself still needs the network to reach its model; EvalGate only reads the files it leaves behind. Prompt, response, tool-argument and tool-output content is dropped while decoding, even when the agent was configured to send it. Each agent profile keeps an allowlist of attributes, so a new content-bearing attribute from a vendor is dropped without a code change. The run has no person in it unless you ask for one with --attribute-by person.

​
1. Capture the agent’s telemetry

Claude Code exports OpenTelemetry over OTLP. Run a local OpenTelemetry Collector with the file exporter and point Claude Code at it. The full collector configuration is in the CLI reference.
For Claude Managed Agents, add a Console telemetry destination instead; see Import Claude Managed Agents runs. For the OpenAI Agents API, save trace export pages; see Import OpenAI Agents API runs. For Gemini Managed Agents, save interactions; see Import Gemini Managed Agents runs. Alternatively, name one Claude Code 2.1 session log with --source claude-session. Claude Code treats that format as internal, so prefer an OTLP capture when the importer reports an unsupported format.

​
2. Import each run

The command previews by default. --apply writes one file per run to .evalgate/runs/<runId>.json. Use --tier to label the depth of each run so views can group by it. Inputs are bounded (file size, line size, records, runs, gzip ratio), and symlinks and FIFOs are refused.

​
3. Score findings against a reviewed answer key

A person writes the answer key: the facts a complete answer must contain, each with the paths or symbols that prove it. Save the agent’s findings as a JSON array of claim, paths and symbols, then score:
Scoring is deterministic and explainable per item. It measures citation coverage against the key. Whether a claim is true is left to the person who reviewed the key.

​
4. Compare runs with a view

Views select measures from a fixed catalog of 110 by id; they hold no formulas. A measure the run cannot supply shows n/a with its reason and is never filled with 0. No measure is labelled good or bad.

​
5. Export the PDF

The report is one page plus a definitions page. It always carries the label “Self-reported, unverified” and a footer saying cost is an estimate. It never contains prompts, responses, code, tool arguments or output, or file paths. --hide-labels replaces titles and run ids with letters.

​
Agent settings that send content

EvalGate drops content at import whatever these settings are. Turn them off at the source too, so content never leaves the developer’s machine in the first place. The Verified column says where each row comes from; rows not confirmed against the vendor’s own documentation say so. Two things follow from the table. Identity attributes arrive even with every content setting off, so EvalGate drops user.email at import and keeps the organization only as a salted digest. And OTEL_LOG_TOOL_DETAILS is the only way Claude Code telemetry carries file paths: EvalGate keeps those paths as salted digests and drops the rest of that payload. Codex CLI, Gemini CLI and Copilot CLI imports are not available yet; their rows are listed so you can turn content off before those profiles ship.