Case Study: Reducing Support Chatbot Errors by 60%

How a SaaS compunknown used systematic evaluation to dramatically improve their AI support agent and increase customer satisfaction by 40%.

The Challenge

TechFlow (name changed), a B2B SaaS platform with 50,000 users, launched an AI support chatbot to handle common customer questions. The goal was ambitious: deflect 60% of support tickets and reduce response time from hours to seconds.

Initial results were promising. The chatbot handled thousands of conversations per week, and customers praised the instant responses. But the support team noticed a troubling pattern: escalations to human agents were increasing, not decreasing.

The problem? The chatbot was confident but often wrong. It would confidently state incorrect information about features, billing, or account settings. Users would try to follow its advice, fail, get frustrated, and escalate to human support—now with two problems instead of one.

The Approach: Systematic Evaluation

TechFlow's engineering team partnered with their support leads to build a comprehensive evaluation system. Here's how they did it:

Phase 1: Understanding the Problem (Week 1-2)

First, they needed to quantify what "good" looked like. The support team manually reviewed 500 chatbot conversations and categorized errors:

• Hallucinations (42%): Made up features that don't exist
• Outdated info (28%): Correct 6 months ago, wrong now
• Misunderstanding intent (18%): Answered the wrong question
• Wrong tone (12%): Too casual or dismissive for enterprise customers

They also created a gold standard test set of 150 real customer questions with expert-reviewed "ideal" responses, covering common scenarios and known failure modes.

Phase 2: Building Evaluation Infrastructure (Week 3-4)

With the test set in hand, they implemented automated evaluation:

They ran their current chatbot through this evaluation: 67/150 test cases failed (55% pass rate). Now they had a baseline to improve against.

Phase 3: Iterative Improvement (Week 5-12)

Armed with data, they systematically addressed each failure category:

Iteration 1: Fix Hallucinations

• Implemented RAG (Retrieval-Augmented Generation) with up-to-date documentation
• Added explicit instruction: "Only answer based on provided context. If unsure, say so."
• Result: Test pass rate improved from 55% → 72%

Iteration 2: Update Knowledge Base

• Audited all documentation for accuracy and freshness
• Added version tags and last-updated dates
• Set up weekly syncs between product and support to flag changes
• Result: Test pass rate improved to 81%

Iteration 3: Better Intent Recognition

• Fine-tuned intent classifier on real support tickets
• Added clarifying questions for ambiguous requests
• Result: Test pass rate improved to 89%

Iteration 4: Tone Refinement

• Updated system prompt with tone examples from top-rated human responses
• Added empathy templates for common frustrations
• Result: Test pass rate improved to 93%

Phase 4: Production Validation (Week 13-16)

With strong eval performance, they rolled out the improved chatbot to 10% of users. They tracked:

• Deflection rate (did users resolve their issue without escalating?)
• Customer satisfaction scores
• Escalation reasons (what still wasn't working?)
• Average resolution time

The A/B test confirmed eval gains translated to production improvements. They expanded to 100% of traffic.

The Results

Key Lessons Learned

1. Measurement enables improvement

Before building evals, the team was flying blind. They knew the chatbot had issues but couldn't quantify them or track progress. Once they had metrics, improvement was straightforward.

2. Start with human expertise

The support team's domain knowledge was critical for building effective rubrics. Engineers alone wouldn't have identified the right evaluation criteria.

3. Iterate in small batches

Rather than one big rewrite, they made targeted improvements and measured impact after each change. This made it easy to identify what worked and what didn't.

4. Offline evals predict production performance

The correlation between eval pass rate and production metrics was strong (r=0.89). This gave the team confidence to ship changes without long A/B tests for every iteration.

5. Evaluation is ongoing, not one-time

TechFlow now runs evals on every code change and continuously expands their test set based on production failures. Evaluation became part of their development workflow, not a separate phase.

What's Next

With a solid foundation, TechFlow is now:

• Expanding to proactive support (predicting issues before users report them)
• Building multilingual support with language-specific eval sets
• Experimenting with different LLM models and comparing performance
• Training a custom judge model on their domain-specific quality standards