Tracing: The Missing Layer in LLM Observability
Why distributed tracing matters for AI applications and what metrics you should be tracking to understand and optimize your LLM systems.
The Observability Gap
Traditional observability tools were built for deterministic systems: API latency, error rates, database queries. These metrics still matter for AI applications, but they don't tell you what you really need to know:
- Why is this LLM call taking 8 seconds when it usually takes 2?
- How munknown tokens am I actually using per request?
- Which part of my RAG pipeline is causing quality issues?
- Did that prompt change reduce latency or just make outputs shorter?
You need tracing—the ability to follow a request through your entire AI pipeline and understand what happened at each step.
What is LLM Tracing?
Tracing captures the full execution path of an AI request as a tree of "spans"—individual operations that make up the larger workflow. For an AI application, that might look like:
📊 Trace: "User asks about pricing"
├─ 🔍 Span: Embed query (45ms, 12 tokens)
├─ 📚 Span: Vector search (123ms, 5 results)
├─ 🤖 Span: LLM call - GPT-5.2
│ ├─ Input: 1,247 tokens
│ ├─ Output: 156 tokens
│ └─ Latency: 2,834ms
└─ ✅ Total: 3,002ms, $0.042 costEach span captures:
- What happened: Operation name and type
- When it happened: Start time and duration
- Inputs/outputs: Prompts, completions, retrieved documents
- Metadata: Model name, temperature, token counts
- Cost: Token usage translated to actual dollars spent
Why Tracing Matters
1. Debug Production Issues
When a user reports "the AI gave me a wrong answer," what do you do? Without tracing, you're guessing. With tracing, you can:
- Pull up the exact trace for that request
- See what prompt was sent to the LLM
- Review what documents were retrieved for context
- Identify if it was a retrieval problem or a generation problem
- Reproduce the issue with the exact inputs
2. Optimize Costs
LLM costs scale with usage, and tokens add up fast. Tracing shows you exactly where money is being spent:
Example Discovery:
A team noticed their LLM costs doubled overnight. Tracing revealed a code change accidentally included the entire conversation history in every request. Average prompt size went from 400 tokens to 2,100 tokens. One line fix saved $15K/month.
3. Improve Latency
Users expect fast responses. Tracing helps you identify bottlenecks:
- Is the LLM call slow, or is retrieval the bottleneck?
- Are you making sequential calls that could be parallel?
- Which model/provider combination is fastest for your use case?
4. Understand Quality Issues
When outputs are wrong, tracing reveals why:
- Did we retrieve the wrong documents? (RAG issue)
- Did we retrieve the right documents but the LLM ignored them? (Prompt issue)
- Did the model hallucinate despite good context? (Model issue)
Key Metrics to Track
Performance Metrics
- End-to-end latency: Total time from request to response
- LLM latency: Just the model inference time
- Time to first token (TTFT): For streaming responses
- Tokens per second: Generation speed
Cost Metrics
- Input tokens: Prompt size
- Output tokens: Completion size
- Cost per request: Total spend per API call
- Cost per user/session: Aggregate spending patterns
Quality Metrics
- Retrieval relevance: Are we finding the right documents?
- Context utilization: Is the LLM using provided context?
- Output diversity: Are responses varied or repetitive?
- Error rates: API failures, timeouts, rate limits
Implementing Tracing
Most AI applications need tracing at three levels:
1. LLM Calls
Capture every LLM API call with full context:
import { trace } from "@/lib/tracing"
const response = await trace.llm({
name: "Generate response",
model: "gpt-5.2-chat-latest",
input: prompt,
metadata: { temperature: 0.7, user_id }
})2. Retrieval Operations
Track what documents are fetched and why:
const docs = await trace.retrieval({
name: "Vector search",
query: userQuery,
results: retrievedDocs,
metadata: { top_k: 5, score_threshold: 0.7 }
})3. Full Request Traces
Wrap the entire request to capture end-to-end behavior:
await trace.request("User question", async () => {
const embedded = await embedQuery(query)
const docs = await vectorSearch(embedded)
const response = await llm.complete(prompt, docs)
return response
})Real-World Use Cases
🔍 Cost Optimization
Identify requests with abnormally high token counts. Found that certain user questions triggered expensive multi-turn conversations. Added smarter conversation pruning, reduced costs by 35%.
⚡ Latency Reduction
Discovered that vector search was taking 400ms while LLM call took 2s. Moved to parallel retrieval and reduced end-to-end latency by 25%.
🐛 Quality Debugging
Users reported "AI doesn't know about recent features." Traces showed retrieval was working, but LLM was ignoring recent docs in favor of general knowledge. Added recency weighting to fix.
Best Practices
- Trace everything in production: You can't debug what you don't capture
- Include user context: User ID, session ID, feature flags help with filtering
- Set up alerts: Notify when latency, cost, or error rates spike
- Sample intelligently: Trace 100% of errors and slow requests, sample the rest
- Link traces to evaluation: Use trace IDs to pull production examples into test sets
- Build dashboards: Monitor trends over time, not just individual traces
Getting Started
If you're not tracing your LLM calls yet, start simple:
- Instrument one critical path (e.g., your main chat endpoint)
- Capture input tokens, output tokens, latency, and cost
- Build a simple dashboard to visualize these metrics over time
- Use traces to debug the next production issue
You'll immediately see patterns and opportunities for optimization that were invisible before.
Start Tracing Your LLM Calls
Our platform provides automatic tracing for all major LLM providers. Get started with our tracing setup guide.
Setup Tracing