Why your chatbot cannot find your own pages

When a chatbot says “I couldn’t find that” about something your website plainly says, the page is usually fine.
The bot looked for it the wrong way.
We found this on our own chatbot in September, and fixing how it searched cut its “couldn’t find” answers from seven to one.

How a chatbot finds the answer

An AI chatbot does not read your whole website for every question.
It cuts your pages into short passages, and when a visitor asks something it fetches the few passages that look most relevant.
It writes its answer from those alone, so if the right passage is not among them, it has nothing to answer from.

Most chatbots pick those passages by meaning.
Each passage becomes a list of numbers that places it near passages about similar things, and the visitor’s question goes through the same step.
That copes well with a visitor who types “price” when your page says “cost”.

Meaning is weakest on the words visitors care about most

Search by meaning works on what a sentence is about.
It works badly on a single word that carries little meaning of its own: a sector, a product name, a town, a price, a reference number.
To that search, a question about an accountancy firm sits close to every page about small businesses, so the one page written for accountants does not stand out.

Researchers have measured this.
A Princeton team found that meaning-based search did far worse than plain keyword search on simple questions about named people, places and things.
The BEIR benchmark, which tested search methods across 18 sets of documents, found plain keyword ranking stayed a hard baseline to beat.

Those single words are what a visitor types.
Nobody asks your chatbot about professional services in general.
They ask about their sector, your product and your price.

What we found on our own chatbot

Our chatbot answers questions about Opafex from the pages of this website.
On 25 September we asked it 15 questions, twice each.
It declined seven of the 30 answers, and two causes accounted for it.

• It read too little. It had room for about 24,000 characters of our site per answer and fetched five passages, roughly 10,000 characters.
  Nobody had chosen five.
  It was a default left over from code written before the room existed.
• It searched by meaning alone. A question that turned on one specific word got passages about the general subject instead.

We changed two things.
The bot now fills the room it has, which means 12 passages instead of five.
It also searches for the exact words in the question and merges those matches with the meaning-based ones.
Up to a quarter of the places go to exact-word matches.

Rarity mattered more than we expected.
Our first version counted matching words, and the misses came back.
Common words in the question appear all over our site, and they outvoted the rare word that named the right page.
Weighting each word by how rare it is across the site fixed that.

The same 15 questions, asked twice the same evening, got 24 answers instead of 20, and one decline instead of seven.
We cannot say how much came from reading more and how much from the exact-word search, because both went live together.

The question we got wrong

One question kept failing: which suits a 20-person accountancy firm, our chatbot or our agent?
We had blamed the search, because the word “accountancy” appears on our site.
The search was fine.
We had a page about accountancy firms and no page that compared the two products, so the bot had nothing to answer from.

The bot answered honestly, and that was the right result.
If your chatbot declines something, check whether a page answers that exact question before you blame the search.
A word appearing on your site does not mean the answer does.

Test your own chatbot in ten minutes

• Ask about names and numbers. Ten questions that each hinge on one specific thing: a product, a sector you serve, a town you cover, a price, an opening day.
• Ask each one twice. Use different words the second time.
  If it answers one wording and not the other, the page exists and the search missed it.
• Read every decline. For each “I couldn’t find that”, open the page you think answers it.
  Either the page answers that exact question and the search failed, or it does not and you have a page to write.
• Ask your supplier two questions. Does the bot search by exact words as well as meaning?
  And how much of your site does it read for each answer?

What a good fix looks like

Anthropic published a clear test of this.
In their contextual retrieval results, giving each passage a line of context cut failed searches by 35%.
Adding exact-word search cut them by 49%, and ranking the results again with a second model cut them by 67%.

You do not need to know how any of that works to buy well.
Ask the supplier how their bot handles names and numbers, then run the ten-minute test on their demo with your own pages.

If you want to see what your own site cannot answer before you buy anything, our free website check asks your pages the questions your customers ask and names the gaps.
Our website chatbot uses both kinds of search described here.