> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Cost of quality for coding agents

> Import a coding agent's own telemetry offline, score its findings against a reviewed answer key, and compare quality against cost in a one-page PDF.

Does a deeper agent run find more of what matters, and what does the extra
depth cost? This workflow answers that for one process in one codebase:
run the agent at a few depths, score each run against an answer key a person
wrote, and lay quality against cost side by side.

> [!WARNING]
>
> This workflow is **Experimental**. It ships in the TypeScript 4.0.0 source
> candidate of `@evalgate/sdk`, which is not published yet, and it is
> TypeScript-only until the Python port. Claude Code is supported, and
> Claude Managed Agents, OpenAI Agents API and Gemini Managed Agents imports
> are Experimental.
> Every number is self-reported by the agent's own telemetry and is not
> verified against gateway or billing data. No hosted view exists yet.

## What EvalGate does and does not do

Everything below runs locally through the CLI. Import, scoring, measures,
views and the PDF never open a network connection, never call a model and
never read a credential. The coding agent itself still needs the network to
reach its model; EvalGate only reads the files it leaves behind.

Prompt, response, tool-argument and tool-output content is dropped while
decoding, even when the agent was configured to send it. Each agent profile
keeps an allowlist of attributes, so a new content-bearing attribute from a
vendor is dropped without a code change. The run has no person in it unless
you ask for one with `--attribute-by person`.

## 1. Capture the agent's telemetry

Claude Code exports OpenTelemetry over OTLP. Run a local OpenTelemetry
Collector with the `file` exporter and point Claude Code at it. The full
collector configuration is in the
[CLI reference](/docs/sdk/cli.md).

```bash theme={null}
export CLAUDE_CODE_ENABLE_TELEMETRY=1
export OTEL_LOGS_EXPORTER=otlp OTEL_METRICS_EXPORTER=otlp
export OTEL_EXPORTER_OTLP_PROTOCOL=http/json
export OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318
claude
```

For Claude Managed Agents, add a Console telemetry destination instead; see
[Import Claude Managed Agents runs](/docs/guides/claude-managed-agents.md). For the
OpenAI Agents API, save trace export pages; see
[Import OpenAI Agents API runs](/docs/guides/openai-agents-api.md). For Gemini
Managed Agents, save interactions; see
[Import Gemini Managed Agents runs](/docs/guides/gemini-managed-agents.md).

Alternatively, name one Claude Code 2.1 session log with
`--source claude-session`. Claude Code treats that format as internal, so
prefer an OTLP capture when the importer reports an unsupported format.

## 2. Import each run

```bash theme={null}
npx @evalgate/sdk harness import --from ./claude-code-otlp.jsonl \
  --run-id deposit-light --tier light --task deposit-trace --commit "$(git rev-parse HEAD)" --apply
```

The command previews by default. `--apply` writes one file per run to
`.evalgate/runs/<runId>.json`. Use `--tier` to label the depth of each run so
views can group by it. Inputs are bounded (file size, line size, records,
runs, gzip ratio), and symlinks and FIFOs are refused.

## 3. Score findings against a reviewed answer key

A person writes the answer key: the facts a complete answer must contain,
each with the paths or symbols that prove it. Save the agent's findings as a
JSON array of `claim`, `paths` and `symbols`, then score:

```bash theme={null}
evalgate harness score --run <runId> --answer-key ./answer-key.json \
  --findings ./findings.json --apply
```

Scoring is deterministic and explainable per item. It measures citation
coverage against the key. Whether a claim is true is left to the person who
reviewed the key.

## 4. Compare runs with a view

```bash theme={null}
evalgate harness measures --group cost
evalgate harness view show --name quality-vs-cost --runs 'deposit-*'
```

Views select measures from a fixed catalog of 110 by id; they hold no
formulas. A measure the run cannot supply shows `n/a` with its reason and
is never filled with `0`. No measure is labelled good or bad.

## 5. Export the PDF

```bash theme={null}
evalgate harness report --view quality-vs-cost --runs 'deposit-*' \
  --title "Deposit trace by depth" --answer-key ./answer-key.json --apply
```

The report is one page plus a definitions page. It always carries the label
"Self-reported, unverified" and a footer saying cost is an estimate. It never
contains prompts, responses, code, tool arguments or output, or file paths.
`--hide-labels` replaces titles and run ids with letters.

## Agent settings that send content

EvalGate drops content at import whatever these settings are. Turn them off
at the source too, so content never leaves the developer's machine in the
first place. The **Verified** column says where each row comes from; rows
not confirmed against the vendor's own documentation say so.

| Agent | Setting | What it sends | Default | Verified |
| - | - | - | - | - |
| Claude Code | `OTEL_LOG_USER_PROMPTS=1` | Prompt text on `user_prompt` | Off | Anthropic docs |
| Claude Code | `OTEL_LOG_ASSISTANT_RESPONSES=1` | Response text (falls back to the prompt setting when unset) | Off | Anthropic docs |
| Claude Code | `OTEL_LOG_TOOL_DETAILS=1` | Tool parameters, Bash commands, file paths, MCP and skill names, error messages | Off | Anthropic docs |
| Claude Code | `OTEL_LOG_TOOL_CONTENT=1` | Tool output on trace span events | Off | Anthropic docs |
| Claude Code | `OTEL_LOG_RAW_API_BODIES` | Full request and response JSON, inline or to files | Off | Anthropic docs |
| Claude Code | No setting; sent when available | `user.email`, `user.account_uuid`, `organization.id`; on events, `workspace.host_paths` | On | Anthropic docs |
| Claude Code | Session log on disk | Prompts, responses, tool input and output | Always written | Third-party descriptions |
| Codex CLI | `log_user_prompt = true` | Prompt text (otherwise `[REDACTED]`) | Off | Third-party guides |
| Codex CLI | `log_agent_responses = true` | Final answers | Off | Third-party guides |
| Codex CLI | `codex.tool_result` event | Arguments and output, as listed in OpenAI's original pull request description | Unknown | Not verified |
| Codex CLI | No setting; reported as always sent | `user.email`, `user.account_id` | On | Third-party guides |
| Gemini CLI | `telemetry.logPrompts` or `GEMINI_TELEMETRY_LOG_PROMPTS` | Prompt text; `request_text` and `response_text` where applicable | Appears to be on; confirm for your version | Google docs extract |
| Copilot CLI | `COPILOT_OTEL_CAPTURE_CONTENT=true` or `captureContent` | Message content | Off | Third-party guides and SDK docs |

Two things follow from the table. Identity attributes arrive even with every
content setting off, so EvalGate drops `user.email` at import and keeps the
organization only as a salted digest. And `OTEL_LOG_TOOL_DETAILS` is the only
way Claude Code telemetry carries file paths: EvalGate keeps those paths as
salted digests and drops the rest of that payload.

Codex CLI, Gemini CLI and Copilot CLI imports are not available yet; their
rows are listed so you can turn content off before those profiles ship.

## Related

* [CLI reference: harness commands](/docs/sdk/cli.md)
* [Import Claude Managed Agents runs](/docs/guides/claude-managed-agents.md)
* [Import OpenAI Agents API runs](/docs/guides/openai-agents-api.md)
* [Import Gemini Managed Agents runs](/docs/guides/gemini-managed-agents.md)
* [Evaluate your subagents](/docs/guides/evaluate-your-subagents.md)
* [Feature status](/docs/platform/feature-status.md)


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.