CONVIRO
Customer Success

Chatbot Analytics: 8 Metrics That Actually Matter

Deflection rate looks great on a slide but hides unhappy customers. These eight chatbot metrics show whether your assistant really solves problems, plus a weekly review routine.

Conviro
September 20, 20267 min read
Chatbot Analytics: 8 Metrics That Actually Matter
In this article

Most chatbot dashboards lead with one flattering number: how many conversations the bot "handled" without a human. It climbs every month, everyone nods, and customers who gave up halfway through a chat are quietly counted as wins.

The chatbot metrics that matter answer a simpler question: did the customer get what they came for, and did the business get value from the conversation? In practice that comes down to eight numbers: resolution rate, handoff rate, CSAT, first response time, fallback rate, lead conversion, answer accuracy and repeat contact rate.

Here's what each one tells you, how to calculate it, and a weekly routine that keeps your assistant improving.

Key takeaways

  • Deflection counts conversations that never reached a human, not problems that got solved. Always pair it with resolution and repeat contact.
  • Track a small set of chatbot KPIs consistently instead of every number your tool can export.
  • A 30-minute weekly transcript review finds more improvements than any dashboard on its own.

Why deflection alone misleads

Deflection (sometimes called containment) is the share of conversations that ended without a human agent. It's easy to measure, which is why it's everywhere.

The problem is what it can't see. A customer who asks about a refund, gets a vague answer and closes the chat counts as deflected. So does someone who gives up on chat and emails you instead. A bot that never offers a human option will post an excellent deflection rate and a poor customer experience.

Treat deflection as a volume indicator, not a success metric.

The 8 chatbot metrics worth tracking

Metric What it tells you Simple formula Warning sign
Resolution rate Problems actually solved Resolved conversations / total High deflection, low resolution
Handoff rate How often people are needed Escalated conversations / total Very low (hidden exit) or rising fast
CSAT for AI chats How customers felt Positive ratings / all ratings AI CSAT well below human CSAT
First response time Speed of the first reply Median time to first reply Long waits after handoff
Fallback rate Gaps in your knowledge "No answer" replies / questions Same gaps every week
Lead conversion Revenue impact Qualified leads / sales chats Leads captured, never followed up
Answer accuracy Trustworthiness Correct answers / reviewed answers Confident but wrong replies
Repeat contact rate Whether answers stuck Same-issue returns / resolved Rising after a content change

1. Resolution rate

The share of conversations where the question was answered or the task completed. Define "resolved" up front: the customer confirmed, rated positively, or didn't return on the same topic within a few days. Without a definition, this number drifts.

2. Handoff rate

How often the assistant passes a conversation to your team. There's no single right level: a clinic or B2B service should expect more handoffs than a store answering shipping questions. Watch the trend and read the reasons, not just the count. Our guide to chatbot-to-human handoff covers when escalation should happen.

3. CSAT for AI conversations

Ask a one-question rating at the end of AI-only conversations and compare it with human-handled ones. A large gap usually points to tone, missing information, or a bot that holds on too long. The CSAT guide covers wording and timing.

4. First response time

AI replies are near-instant, so the interesting number is what happens after handoff. If customers wait two hours after the bot says "Let me connect you," the experience is slow. Track first response time for the bot and the human queue separately.

5. Fallback ("no answer") rate

How often the assistant says it can't help or gives a generic reply. Group these by topic. A cluster of unanswered questions about international shipping is a content task for this week, not a mystery.

6. Lead conversion from chat

How many sales conversations produced a qualified lead, and how many of those became a booking, demo or order. If you collect contact details but nobody follows up quickly, the chatbot isn't the bottleneck.

7. Answer accuracy

Grade a sample of AI answers as correct, partly correct or wrong against your own sources. This protects your brand: a confidently wrong answer about a return policy costs more than no answer.

8. Repeat contact rate

Customers who come back about the same issue within a set window, such as seven days. Resolution rate shows obvious failures; repeat contacts catch the hidden ones.

How to measure chatbot performance in a weekly review

Dashboards tell you where to look. Transcripts tell you what to fix. Block 30 minutes a week and follow the same routine:

  1. Compare the trend for resolution, handoff, CSAT and fallback rate against last week.
  2. Read 15–20 transcripts, mixing low-rated chats, handoffs and a few random ones.
  3. Grade accuracy on those samples with the correct / partly correct / wrong scale.
  4. List the top three unanswered topics from fallback replies.
  5. Fix content, not just settings: add or update the knowledge source behind each gap.
  6. Review handoff reasons with whoever works the human queue. Was the timing right, and was there enough context?
  7. Log one change and check its effect next week.

After a month, your log shows which changes moved the numbers and which didn't.

Chatbot KPIs and targets: start from your own baseline

Resist copying a target from someone else's slide deck. Your question mix, industry and knowledge base decide what's realistic.

Measure for two to four weeks, then set goals relative to that baseline. For example, if you handle 1,200 conversations a month and 180 end in "I don't know," a sensible goal is to close the five biggest content gaps behind those 180, not to chase an arbitrary percentage.

Segment where you can: by channel (website chat, WhatsApp, Instagram), by topic and by language.

How Conviro supports chatbot analytics

Conviro's AI Assistants answer from your own knowledge base, and the built-in analytics cover the numbers above that matter most: CSAT, resolution rates and AI accuracy. When a conversation is handed off, it lands in the Live Inbox with the transcript, detected intent, tags and collected details, which makes reviewing handoff reasons quick.

If you run several brands, the multi-brand workspace keeps each assistant's knowledge base and analytics separate, so one brand's numbers don't hide another's problems.

Frequently Asked Questions

What is a good resolution rate for a chatbot?

There's no universal benchmark worth trusting, because resolution depends on your question mix and how complete your knowledge base is. A store answering shipping and returns questions can resolve far more than a clinic handling sensitive scheduling. Measure your own baseline for a few weeks, then improve it by closing the biggest content gaps.

What's the difference between containment and resolution?

Containment, or deflection, counts conversations that never reached a human. Resolution counts conversations where the customer's problem was actually solved. A chat can be contained but unresolved, for example when a frustrated customer simply leaves. Track both numbers and treat a widening gap between them as a warning sign.

How many chatbot conversations should I review each week?

For most small and mid-sized teams, 15 to 20 transcripts a week is enough to spot patterns without becoming a chore. Mix low-rated conversations, handoffs and a few random samples. After a big knowledge base change, review more for two weeks.

Which chatbot KPIs should I report to leadership?

Keep it to four: resolution rate, CSAT for AI conversations, lead conversion from chat and answer accuracy from your weekly sample. Together they show customer outcome, experience, revenue impact and risk. Add one line on the biggest change you made that month and its effect.

Next steps

Pick four of these eight metrics, record a baseline this week, and put a recurring 30-minute review in your calendar. If you want an assistant whose analytics and transcripts make that review easy, start free with Conviro.

Share:

Published by

Conviro

Sharing insights on AI-powered customer support, chatbot automation, and SaaS growth strategies.

Put the playbook into practice

Turn your support knowledge into faster answers

Explore Conviro’s AI support workspace, then start free when the workflow fits your team.

Related articles

More Posts