How to Test AI Chatbot Answers with BetterDocs Test Bench

BetterDocs Test Bench lets you check how well your AI Chatbot answers real customer questions before your visitors ever ask them. It runs a set of questions against the chatbot, gives you a pass score, and shows a clear reason for every failed answer, such as Wrong doc or Refused. This way, you can spot weak answers, fix the docs behind them, and make sure your chatbot gives accurate replies every time.

Примечание: Make sure the BetterDocs AI Chatbot is already set up and your docs are synced. If you haven’t done that yet, follow this guide on how to configure BetterDocs AI Chatbot.

Step 1: Navigate to Test Bench #

From your WordPress dashboard, go to wp-admin → BetterDocs → AI Chatbot. From the left panel of the AI Chatbot settings, click on ‘Test Bench’.

BetterDocs AI Chatbot settings with Test Bench open from the left panel

Here you will see the current test status, the number of questions in your set, and an estimated cost for the next run. You will also find three tabs: ‘Results’, ‘Questions’ а также ‘Run history’.

Step 2: Build Your Question Set #

Click on the ‘Questions’ tab. This is where you create the list of questions Test Bench will ask your chatbot. You can build the set in several ways, and you can combine them.

Step 2.1. Get AI Suggestions from Your Docs #

Click ‘Suggest from docs’. With one click, AI reads your docs and suggests 20 questions your visitors are likely to ask. Each suggested question is tagged as ‘Suggested’ and is automatically linked to the doc that should answer it.

Step 2.2. Add Questions Manually #

Paste your own questions in the ‘Paste questions, one per line’ field and click ‘Add’. This is useful when you already know the questions your customers ask most often.

Step 2.3. Import Questions from a CSV File #

Click ‘Import CSV’ to upload questions in bulk. Your CSV file should have these columns:

question: the question you want to test
expected_doc: the doc that should answer it (ID, URL, slug or title)
expected_answer: the answer you expect the chatbot to give

Step 2.4. Import from Chat History #

Click ‘Import from chat history’ to add the 50 most recent visitor questions from your real chatbot conversations. This helps you test the chatbot with questions people have actually asked.

Step 2.5. Fine-Tune Each Question #

Under each question, you can adjust how it is checked:

Expected doc: shows the doc linked to the question. Click the × icon to remove it.
‘+ Expected answer’: add the answer you expect, so the Test Bench can compare it with the chatbot’s reply.
‘+ Image’: attach an image to the question, for example a product photo a visitor might share.
‘Remove’: delete the question from your set.

BetterDocs Test Bench Questions tab to build and fine-tune the question set

Step 3: Run the Test #

Before you start, check the cost estimate on the status card. Test Bench shows the approximate token usage and cost on your connected AI model key before every run, so you always know what a test will cost. The estimate gets more accurate after your first run.

When you are ready, click the ‘Run [number] questions’ button. Test Bench will start asking your chatbot each question in the set.

Running a BetterDocs Test Bench question set against the AI Chatbot

Test runs don’t count toward your chat history or analytics, so your real visitor data stays clean.

Step 4: Review the Results #

Once the run finishes, the status card shows your pass score along with how many questions passed, for example ’18 of 20 questions passed’.

Перейти к ‘Results’ tab to see each question’s outcome. You can filter the list by:

All: every question in the set
Failed: questions the chatbot answered incorrectly
Passed: questions answered correctly
Not run: questions that were skipped, cancelled or errored

For every failed question, Test Bench shows the reason, such as Wrong doc (the chatbot pulled from the wrong doc) or Refused (the chatbot didn’t answer). Use these reasons to find which docs need to be updated or added.

BetterDocs Test Bench Results tab showing pass score and failed answer reasons

Step 5: Track Your Run History #

Click the ‘Run history’ tab to see every previous test. For each run, you will find:

When: the date and time of the run
Why: what started the run, for example ‘Run by hand’
Score: the pass score, or the run status if it was cancelled or failed
Passed: the number of questions passed out of the total
Model: the AI model used for the run

Click ‘View’ next to any run to see its full results. This helps you compare scores over time and check whether your doc updates improved the chatbot’s answers.

BetterDocs Test Bench Run history tab with scores of previous test runs

Automatic Re-Test After Re-Sync #

You don’t have to remember to re-test after big content changes. Test Bench automatically re-runs your question set after a full re-sync of your docs. If your score drops by 5 points or more, it warns you, so you can catch problems right away.

Окончательный результат #

Your AI Chatbot now has a ready-made test set, a pass score and clear failure reasons. You can find and fix bad answers before your visitors ever see them.