CONVIRO
Guides & Tutorials

AI Chatbot Testing Checklist: 20 Things to Check Before You Go Live

A 20-point chatbot testing checklist covering knowledge, edge cases, languages, handoff, lead capture, privacy, every channel and mobile, so launch day holds no surprises.

Conviro
October 6, 20266 min read
AI Chatbot Testing Checklist: 20 Things to Check Before You Go Live
In this article

The first real customer to talk to your chatbot will ask something nobody on your team thought to try. They'll misspell the product name, ask two questions at once, switch to German halfway through, and want a human at 11 pm.

Chatbot testing is how you meet that customer before launch instead of after. In practice, it means running your assistant through real questions, awkward edge cases and every channel it will live on, then fixing what breaks before you go live.

Below is a 20-point checklist, grouped by area. Plan a focused afternoon with two or three colleagues.

Key takeaways

  • Test with real questions from your inbox history, not the ones you wish customers asked.
  • Handoff, privacy and channel behavior break more launches than wrong answers do.
  • Log every failed check and re-run the whole list after fixes.

Set up the test session

Pull 30 to 50 real customer questions from your email, chat and social inboxes. Give each tester a role: one plays a confused shopper, one an annoyed customer, one tries to break things. Log every check in a simple table:

Check Expected result Actual result Pass/fail Fix owner
"Can I return a sale item?" States final-sale policy, links returns page Said 30-day returns Fail Support lead

Knowledge coverage

  1. Your top 20 questions get correct answers. Compare each against current policies, not memory.
  2. Every knowledge source actually loaded. A PDF that didn't process looks like a "bad AI" problem when it's really missing content. Conviro's new knowledge upload and training-mode reports show how your uploaded sources were processed, so review them before anyone starts asking questions.
  3. Prices, hours and policies are current. Remove old price lists so they can't compete with new ones.
  4. Unknown questions get an honest answer. Ask something you never documented. The assistant should say it doesn't know and offer a next step, not improvise.

Edge cases

  1. Messy input works. Typos, slang, all caps, one-word messages like "refund?".
  2. Two questions get two answers. "Do you ship to Austria and how long does it take?"
  3. Manipulative prompts fail. Ask for a discount it can't give, ask about competitors, type "ignore your instructions". The assistant should stay inside its boundaries.

Languages

  1. Every market language works, including a switch mid-conversation.
  2. Questions in other languages still find the answer. If your knowledge base is in English and a customer writes in Spanish, the answer should be correct and in Spanish. Check the widget's interface language too.

Handoff

  1. "I want a human" works everywhere. Try several phrasings: "agent", "real person", "stop the bot".
  2. Sensitive topics escalate. Complaints, legal threats and safety issues should reach a person. Our handoff guide covers which triggers to set.
  3. Contact details are captured before handoff. If nobody is online, the assistant should collect a name and email or number and say honestly when the team will reply.

Lead capture

  1. Lead details arrive where sales works. Complete a test inquiry and confirm the name, need and timing appear in your CRM or alert channel.
  2. The assistant doesn't interrogate. It should answer first and ask for details when they're useful, not open with a form.

Privacy

  1. Customers know they're talking to AI. Article 50 of the EU AI Act sets transparency obligations for AI systems that interact with people, so make sure the greeting says clearly that it's an AI assistant.
  2. No data leaks. Ask about another customer's order and ask it to repeat internal notes. Confirm it never requests sensitive data it doesn't need.

Every channel

  1. Each channel gets its own test. Web chat, WhatsApp and Instagram DM display links, line breaks and long answers differently. Read the replies on each.
  2. Handoffs from every channel reach the inbox. Escalate once per channel and confirm the conversation appears in your shared inbox and someone is notified.

Mobile

  1. The widget behaves on small screens. It shouldn't cover the checkout button or cookie banner, and the keyboard shouldn't hide the input field.
  2. Answers are phone-sized. If a reply needs scrolling, shorten the source content. Our guide to mobile chat UX has more detail.

After go-live, keep testing

A launch checklist catches the obvious failures; real conversations reveal the rest. For the first two weeks, read a sample of conversations daily, then switch to a weekly review with a scoring rubric like the one in our guide to measuring chatbot accuracy.

Keep your failed checks as a regression list and re-run it whenever you add a large document, change a policy or open a new channel. Conviro's AI assistants also surface knowledge gaps, the questions the assistant couldn't answer well, which tells you what to write next. Setup instructions for each step are in the help center.

Frequently Asked Questions

How long does chatbot testing take before launch?

For a small or mid-sized business, a focused afternoon covers this checklist, plus time to fix what fails and re-test. Plan a shorter second session a day or two later. Several languages or channels add time, so budget roughly an extra hour per language and per channel on top of the core run.

Who should test an AI chatbot before launch?

People who know your customers: support agents, a salesperson and someone who knows the policies. Add one person who has never seen the setup, because they'll phrase questions like a real customer. Avoid relying only on whoever built the knowledge base; they tend to ask questions the way the documents are written.

What should I do when the chatbot fails a check?

Find the cause before changing anything. Most failures come from missing, outdated or conflicting content, so fix the source document first. If the content is right but the answer isn't, review guardrails and handoff rules. Then re-run the failed check plus a few related ones, since a fix in one place can change answers elsewhere.

Can I test a chatbot without customers seeing it?

Yes, and you should. Test on a staging page or a page that isn't linked from your navigation before the widget goes on your main site. For WhatsApp and Instagram, run your first tests from team members' personal accounts before you announce the channel. Only promote the chatbot once the full checklist passes.

Ready for launch day

When all 20 checks pass, go live on one channel first, watch it for a few days, then add the next. Start free with Conviro to build your assistant and run this checklist against it.

Share:

Published by

Conviro

Sharing insights on AI-powered customer support, chatbot automation, and SaaS growth strategies.

Put the playbook into practice

Turn your support knowledge into faster answers

Explore Conviro’s AI support workspace, then start free when the workflow fits your team.

Related articles

More Posts