> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Traces

# Traces: visibility into LLM behavior

> Learn what traces and spans are, what data they capture, how sampling works, and how to promote real failures into evaluation test cases.

A trace is a structured record of one complete run of your AI system — everything that happened from the moment a request arrived to the moment a response was returned. Traces give you the ground truth you need to understand what your AI is actually doing in production, not what you expect it to do in tests.

## Traces and spans

A **trace** represents the whole workflow. A **span** represents one individual step inside that workflow — a single LLM call, a tool invocation, a retrieval operation, or any other discrete unit of work.

A simple chatbot request might produce one trace with one span. A RAG pipeline might produce one trace with four spans: embed the query, retrieve documents, re-rank results, and generate the response. The trace holds the end-to-end picture; spans let you isolate latency, cost, and correctness at each step.

## What gets captured

Every trace and span records:

| Field        | What it contains                                |
| ------------ | ----------------------------------------------- |
| **Input**    | The full prompt or query sent to the model      |
| **Output**   | The full response returned                      |
| **Tokens**   | Input token count, output token count           |
| **Latency**  | Duration in milliseconds                        |
| **Cost**     | Estimated cost based on model pricing           |
| **Model**    | Model name, version, and provider               |
| **Metadata** | User ID, session ID, feature flags, custom tags |
| **Errors**   | Stack traces and error messages when calls fail |

## Instrumenting your application

Use the SDK to create traces and attach spans wherever your application calls an LLM or performs a step you want to observe.

<CodeGroup>
  ```ts TypeScript theme={null} theme={null}
  import { AIEvalClient } from '@evalgate/sdk';

  const client = AIEvalClient.init({
    apiKey: process.env.EVALGATE_API_KEY
  });

  // Create a trace for the full workflow
  const trace = await client.traces.create({
    name: 'Customer Support Query',
    traceId: 'trace-' + Date.now(),
    metadata: { userId: 'user_123', sessionId: 'session_456' }
  });

  // Add a span for the LLM call
  await client.traces.createSpan(trace.id, {
    name: 'LLM Call',
    type: 'llm',
    input: userQuery,
    output: response,
    metadata: { model: 'gpt-5.2-chat-latest', tokens: 150 }
  });
  ```

  ```python Python theme={null} theme={null}
  from evalgate_sdk import AIEvalClient
  from evalgate_sdk.types import CreateTraceParams, CreateSpanParams
  import time

  client = AIEvalClient.init()  # reads EVALGATE_API_KEY env var

  # Create a trace for the full workflow
  trace = await client.traces.create(CreateTraceParams(
      name='Customer Support Query',
      trace_id=f'trace-{int(time.time() * 1000)}',
      metadata={'userId': 'user_123', 'sessionId': 'session_456'}
  ))

  # Add a span for the LLM call
  await client.traces.create_span(trace.id, CreateSpanParams(
      name='LLM Call',
      type='llm',
      input=user_query,
      output=response,
      metadata={'model': 'gpt-5.2-chat-latest', 'tokens': 150}
  ))
  ```
</CodeGroup>

### Multi-step workflows

For pipelines with multiple LLM calls or tool steps, attach one span per step to the same trace. This lets you see the full timeline and pinpoint exactly where latency or quality problems occur.

<CodeGroup>
  ```ts TypeScript theme={null} theme={null}
  import { WorkflowTracer, traceWorkflowStep } from '@evalgate/sdk';

  const tracer = new WorkflowTracer(client);

  await tracer.startWorkflow('RAG Pipeline');

  const embedding = await traceWorkflowStep(tracer, 'embed-query', async () => {
    return await openai.embeddings.create({...});
  });

  const docs = await traceWorkflowStep(tracer, 'retrieve-docs', async () => {
    return await vectorDb.search(embedding);
  });

  const response = await traceWorkflowStep(tracer, 'generate-response', async () => {
    return await openai.chat.completions.create({...});
  });

  await tracer.endWorkflow({ status: 'success' });
  ```

  ```python Python theme={null} theme={null}
  from evalgate_sdk import WorkflowTracer
  from evalgate_sdk.workflows import trace_workflow_step

  tracer = WorkflowTracer(client)

  await tracer.start_workflow("RAG Pipeline")

  embedding = await trace_workflow_step(
      tracer, "embed-query",
      lambda: openai.embeddings.create(...)
  )

  docs = await trace_workflow_step(
      tracer, "retrieve-docs",
      lambda: vector_db.search(embedding)
  )

  response = await trace_workflow_step(
      tracer, "generate-response",
      lambda: openai.chat.completions.create(...)
  )

  await tracer.end_workflow({"status": "success"})
  ```
</CodeGroup>

## Asymmetric sampling

For high-volume applications, analyzing every successful request can be expensive and unnecessary. EvalGate uses asymmetric sampling for server-side failure analysis by default:

* **10%** of successful traces are queued for analysis
* **100%** of error traces and thumbs-down feedback traces are queued for analysis

This controls analysis work after traces are ingested. SDK collector helpers send traces by default unless you configure client-side sampling.

<Tip>
  Always send full traces during development and staging. Reserve client-side sampling for production once you have a baseline sense of your traffic patterns and failure rate.
</Tip>

## Enriching traces with metadata

Traces become much more useful when they carry business context alongside the model inputs and outputs. Attach metadata at trace creation time to enable filtering, grouping, and alerting in the dashboard.

<CodeGroup>
  ```ts TypeScript theme={null} theme={null}
  await tracer.startWorkflow('content-generation', undefined, {
    userId: user.id,
    contentType: 'blog-post',
    targetAudience: 'developers',
    keywords: ['AI', 'evaluation', 'testing']
  });
  ```

  ```python Python theme={null} theme={null}
  await tracer.start_workflow("content-generation", metadata={
      "userId": user.id,
      "contentType": "blog-post",
      "targetAudience": "developers",
      "keywords": ["AI", "evaluation", "testing"]
  })
  ```
</CodeGroup>

<Warning>
  Never log sensitive PII inside trace metadata without proper anonymization. EvalGate scrubs PII before sending traces to external judge providers, but raw metadata is stored as-is. Apply anonymization at the application layer before calling `createTrace` or `startWorkflow`.
</Warning>

## Viewing traces in the dashboard

Once your application is instrumented, open the **Traces** page in your EvalGate dashboard to:

* Search and filter traces by metadata, tags, model, or time range
* View detailed timelines showing nested spans and their durations
* Analyze token usage and cost breakdowns per step
* Inspect full input and output text for any span
* Debug failures with complete error stack traces
* Identify performance bottlenecks across the workflow

## From traces to evaluations

The most important thing you can do with a traced failure is convert it into a permanent test case. EvalGate's label command walks you through every captured trace and lets you mark each one pass or fail, assign a failure mode, and add it to your golden dataset.

```bash theme={null} theme={null}
# Label production traces interactively
npx @evalgate/sdk label
# Arrow-key menu, u to undo, Ctrl-C saves progress

# See failure-mode frequency across all labeled traces
npx @evalgate/sdk analyze
```

Once labeled and promoted into an eval suite, those traces can become evaluation cases that run on every code change. Real production failures become regression coverage after review or promotion.

See [Evaluations](/docs/concepts/evaluations) for how to build test suites from your labeled traces, and [The trace → eval → gate workflow](/docs/concepts/trace-eval-gate) for how the full loop fits together.