Integrate EvalGate with LangChain workflows

Add distributed tracing and evaluations to LangChain chains, agents, and RAG pipelines to monitor quality and catch regressions in production.

LangChain makes it easy to build complex LLM pipelines, but that complexity introduces more failure points — a broken tool, a retrieval miss, or a degraded prompt can silently reduce quality across thousands of requests. Wrapping your LangChain components with EvalGate tracing gives you end-to-end visibility into every step and lets you run structured evaluations against known-good baselines. This guide covers setup, tracing common LangChain patterns, running evaluations against chains, and monitoring production workflows.

Install dependencies

npm install @evalgate/sdk langchain openai

pip install "evalgate-sdk[langchain]" langchain openai

Add your credentials to .env:

.env

EVALGATE_API_KEY=sk_test_your_api_key_here
EVALGATE_ORGANIZATION_ID=00000000-0000-4000-8000-000000000001
OPENAI_API_KEY=your_openai_key

Initialize the SDK

import { AIEvalClient, WorkflowTracer } from '@evalgate/sdk'

const client = AIEvalClient.init()
const tracer = new WorkflowTracer(client)

from evalgate_sdk import AIEvalClient, WorkflowTracer

client = AIEvalClient.init()
tracer = WorkflowTracer(client)

Tracing LangChain components

Simple chains

Wrap your chain call in a WorkflowTracer workflow and create spans for each step:

import { LLMChain } from 'langchain/chains'
import { traceWorkflowStep } from '@evalgate/sdk'

await tracer.startWorkflow('Product Description Chain', undefined, {
  productId: 'prod_123'
})

const result = await traceWorkflowStep(tracer, 'LLMChain', async () => {
  const chain = new LLMChain({ llm, prompt })
  return await chain.call({ product: 'laptop' })
})

await tracer.endWorkflow({ status: 'success' })

from langchain.chains import LLMChain

async def run_chain():
    await tracer.start_workflow('Product Description Chain', metadata={'product_id': 'prod_123'})

    span = await tracer.start_agent_span('LLMChain', input={'product': 'laptop'})
    chain = LLMChain(llm=llm, prompt=prompt)
    result = chain.run(product='laptop')
    await tracer.end_agent_span(span, output={'text': result})

    await tracer.end_workflow(output={'status': 'success'})
    return result

Agents with tool use

Use traceWorkflowStep to wrap each agent invocation so tool calls appear as named spans:

import { initializeAgentExecutorWithOptions } from 'langchain/agents'
import { traceWorkflowStep } from '@evalgate/sdk'

const executor = await initializeAgentExecutorWithOptions(tools, llm, {
  agentType: 'zero-shot-react-description'
})

await tracer.startWorkflow('research-agent', undefined, { query })

const result = await traceWorkflowStep(tracer, 'AgentExecution', async () => {
  return await executor.call({ input: query })
})

await tracer.endWorkflow({ status: 'success' })

from langchain.agents import initialize_agent, AgentType

async def run_agent(query: str):
    agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION)

    await tracer.start_workflow("research-agent", metadata={"query": query})

    span = await tracer.start_agent_span("AgentExecution", input={"query": query})
    result = agent.run(query)
    await tracer.end_agent_span(span, output={"result": result})

    await tracer.end_workflow(output={"status": "success"})
    return result

RAG pipelines

For multi-step RAG pipelines, use traceWorkflowStep to create separate spans for embedding, retrieval, and generation:

import { traceWorkflowStep } from '@evalgate/sdk'

async function ragQuery(question: string) {
  await tracer.startWorkflow('documentation-qa', undefined, { question })

  const embedding = await traceWorkflowStep(tracer, 'embed-query', async () => {
    return await openai.embeddings.create({ model: 'text-embedding-3-small', input: question })
  })

  const docs = await traceWorkflowStep(tracer, 'retrieve-docs', async () => {
    return await vectorstore.similaritySearch(question, 4)
  })

  const answer = await traceWorkflowStep(tracer, 'generate-answer', async () => {
    return await qaChain.call({ query: question, documents: docs })
  })

  await tracer.endWorkflow({ status: 'success' })
  return answer
}

async def rag_query(question: str):
    await tracer.start_workflow("documentation-qa", metadata={"question": question})

    embed_span = await tracer.start_agent_span("embed-query", input={"question": question})
    embedding = embeddings.embed_query(question)
    await tracer.end_agent_span(embed_span, output={"dimensions": len(embedding)})

    retrieve_span = await tracer.start_agent_span("retrieve-docs", input={"question": question})
    docs = vectorstore.similarity_search(question, k=4)
    await tracer.end_agent_span(retrieve_span, output={"doc_count": len(docs)})

    gen_span = await tracer.start_agent_span("generate-answer", input={"doc_count": len(docs)})
    answer = qa_chain.run(question)
    await tracer.end_agent_span(gen_span, output={"answer": answer})

    await tracer.end_workflow(output={"status": "success"})
    return answer

Multi-turn conversations with memory

Group a full conversation session as a single workflow, with one span per turn:

import { ConversationChain } from 'langchain/chains'
import { BufferMemory } from 'langchain/memory'

const memory = new BufferMemory()
const conversation = new ConversationChain({ llm, memory })

await tracer.startWorkflow('multi-turn-conversation', undefined, {
  sessionId: session_id
})

async function chat(message: string) {
  const span = await tracer.startAgentSpan('turn', { input: message })
  const response = await conversation.call({ input: message })
  await tracer.endAgentSpan(span, { output: response.response })
  return response.response
}

await chat('Hello!')
await chat("What's the weather like in Paris?")

await tracer.endWorkflow({ status: 'success' })

from langchain.memory import ConversationBufferMemory
from langchain.chains import ConversationChain

memory = ConversationBufferMemory()
conversation = ConversationChain(llm=llm, memory=memory)

async def run_conversation(session_id: str):
    await tracer.start_workflow("multi-turn-conversation", metadata={"session_id": session_id})

    async def chat(message: str):
        span = await tracer.start_agent_span("turn", input={"message": message})
        response = conversation.predict(input=message)
        await tracer.end_agent_span(span, output={"response": response})
        return response

    await chat("Hello!")
    await chat("What's the weather like in Paris?")

    await tracer.end_workflow(output={"status": "success"})

Running evaluations against chains

Write eval test cases

Define test cases for your chain with createTestSuite, pass chain outputs through the executor, and assert quality with built-in assertions:

import { createTestSuite, expect } from '@evalgate/sdk'
import { LLMChain } from 'langchain/chains'

const chain = new LLMChain({ llm, prompt })

const suite = createTestSuite('Blog Generator Quality', {
  executor: async (topic: string) => {
    const result = await chain.call({ topic })
    return result.text
  },
  cases: [
    {
      input: 'machine learning',
      assertions: [
        (output) => expect(output).toHaveLength({ min: 100, max: 2000 }),
        (output) => expect(output).toContainKeywords(['machine learning']),
        (output) => expect(output).toHaveProperGrammar(),
        (output) => expect(output).toNotContainPII(),
      ]
    },
    {
      input: 'cooking recipes',
      assertions: [
        (output) => expect(output).toHaveLength({ min: 100, max: 2000 }),
        (output) => expect(output).toContainKeywords(['recipe']),
      ]
    }
  ]
})

const results = await suite.run()
console.log(`Pass rate: ${results.passed}/${results.total}`)

from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig
from langchain.chains import LLMChain

chain = LLMChain(llm=llm, prompt=prompt)

async def run_chain(topic: str) -> str:
    result = chain.run(topic=topic)
    return result

suite = create_test_suite('Blog Generator Quality', TestSuiteConfig(
    evaluator=run_chain,
    test_cases=[
        TestSuiteCase(
            name='machine-learning',
            input='machine learning',
            assertions=[
                {'type': 'length', 'min': 100, 'max': 2000},
                {'type': 'contains', 'value': 'machine learning'},
                {'type': 'not_contains_pii'},
            ],
        ),
    ],
))

result = await suite.run()
print(f"Pass rate: {result.passed_count}/{result.total}")

Gate regressions in CI

Once your test suite is defined, add a gate step so every code change is compared against the baseline:

npx @evalgate/sdk gate --format github

Or use the full CI command that handles discovery, baseline comparison, and PR annotations automatically:

.github/workflows/evalgate.yml

name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx @evalgate/sdk ci --format github --write-results --base main
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}

Monitoring production chains

Add rich metadata

Include request context in workflow metadata to enable filtering and debugging in the dashboard:

await tracer.startWorkflow('customer-support-chain', undefined, {
  userId: user.id,
  sessionId: session.id,
  intent: detectedIntent,
  contextLength: conversationHistory.length
})

await tracer.start_workflow("customer-support-chain", metadata={
    "user_id": user.id,
    "session_id": session.id,
    "intent": detected_intent,
    "context_length": len(conversation_history)
})

Tracing strategy by level

High level
Mid level
Low level

Trace the entire chain as a single workflow for end-to-end monitoring. Best for production health checks and cost tracking.

Label production traces for your golden dataset

After collecting production traces, use the CLI to label them interactively and build evaluation coverage from real failures:

# Label unlabeled traces one by one
npx @evalgate/sdk label

# See failure-mode frequency across labeled traces
npx @evalgate/sdk analyze

Sample traces for high-throughput applications — trace 10% of requests to keep overhead low while retaining full error visibility. EvalGate samples 100% of error traces by default.

Best practices

Name spans after steps

Use descriptive span names like embed-query and retrieve-docs instead of generic names like step-1. Specific names make timeline debugging much faster.

Attach relevant metadata

Include userId, sessionId, and model version in workflow metadata so you can filter traces by user segment or model version in the dashboard.

Test at each layer

Test retrieval, generation, and end-to-end quality separately. A passing end-to-end score can mask a broken retrieval step.

Promote failures to tests

When a production chain produces a bad output, capture that input as a test case in your eval suite so the same failure cannot recur.

Troubleshooting

Traces not appearing in the dashboard? Confirm the SDK is initialized with the correct EVALGATE_API_KEY and that WorkflowTracer is instantiated before any workflow calls. Spans are missing or out of order? Make sure every async call inside a traceWorkflowStep callback is properly await-ed. Unawaited promises can resolve after the span closes, causing incomplete data. High latency overhead? The SDK adds roughly 10ms of overhead per trace upload. Use enableBatching: true when initializing the client to group writes into fewer API calls.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Langchain integration

Integrate EvalGate with LangChain workflows

Install dependencies

Initialize the SDK

Tracing LangChain components

Simple chains

Agents with tool use

RAG pipelines

Multi-turn conversations with memory

Running evaluations against chains

Write eval test cases

Gate regressions in CI

Monitoring production chains

Add rich metadata

Tracing strategy by level

Label production traces for your golden dataset

Best practices

Name spans after steps

Attach relevant metadata

Test at each layer

Promote failures to tests

Troubleshooting

​Integrate EvalGate with LangChain workflows

​Install dependencies

​Initialize the SDK

​Tracing LangChain components

​Simple chains

​Agents with tool use

​RAG pipelines

​Multi-turn conversations with memory

​Running evaluations against chains

​Write eval test cases

​Gate regressions in CI

​Monitoring production chains

​Add rich metadata

​Tracing strategy by level

​Label production traces for your golden dataset

​Best practices

Name spans after steps

Attach relevant metadata

Test at each layer

Promote failures to tests

​Troubleshooting

Integrate EvalGate with LangChain workflows

Install dependencies

Initialize the SDK

Tracing LangChain components

Simple chains

Agents with tool use

RAG pipelines

Multi-turn conversations with memory

Running evaluations against chains

Write eval test cases

Gate regressions in CI

Monitoring production chains

Add rich metadata

Tracing strategy by level

Label production traces for your golden dataset

Best practices

Troubleshooting