What an AI Chatbot Can Actually Answer (and Where It Quietly Fails)

“Can we train it on our data?” is the first question I get about AI assistants, and the answer is usually no — and that’s good news, because what actually happens is cheaper, faster to update, and easier to control.

How it really works

Almost every business chatbot worth building uses a pattern called retrieval-augmented generation, usually shortened to RAG. In plain terms:

1. Your content — help pages, policies, product docs, past support replies — gets chopped into chunks and indexed.
2. A customer asks a question.
3. The system searches that index for the handful of chunks most relevant to the question.
4. Those chunks get handed to the language model along with the question and an instruction: answer using only this.

The model isn’t remembering your business. It’s reading the relevant pages at the moment it answers. Which means updating your chatbot is just updating your content — no retraining, no waiting, no cost. Change the refund policy page, and the bot’s answer changes with it.

This is also why “training on your data” is the wrong mental model to buy against. Actual fine-tuning teaches a model a style or a format, not facts — and facts are what your customers are asking for. Anyone quoting you a large number to “train a model on your business” is either describing RAG in misleading language or selling you something you don’t need.

Where it works genuinely well

• Questions with a written answer. Shipping times, warranty terms, opening hours, what’s included in a package, how to reset a password.
• The same 30 questions you answer every week. Go look at your inbox. Most support volume is a short list repeated forever.
• Documented process explanations. “How do I return an item?” — as long as the process is actually documented somewhere.
• Multilingual answering. One well-written English knowledge base can answer in a dozen languages surprisingly well.
• Out-of-hours cover. Often the strongest business case of all — not replacing your team, but answering the 11pm question that would otherwise wait until morning.

Where it fails, often confidently

Live data it can’t see. “Where’s my order?” is the most-asked question in eCommerce and RAG cannot answer it — that information is in your order system, not your content. This is solvable, but by connecting the bot to your order API as a tool, not by adding more documents. Treat it as a separate build with its own budget.

Anything requiring calculation. Custom quotes, tax, shipping across weight bands. Language models are unreliable at arithmetic and worse at applying pricing rules with exceptions. Give it a calculator or a lookup, or route the question to a human.

Judgement calls and exceptions. “The item arrived damaged but I threw away the box, can I still return it?” The honest answer lives in someone’s head, not your policy page. The bot will invent something reasonable-sounding. That’s worse than saying nothing.

Questions whose answer isn’t written down anywhere. This is the real limit, and it’s not an AI problem. If your team knows something that exists in no document, the bot cannot know it. Every chatbot project is partly a documentation project — that’s the part people don’t budget for.

Your content is the ceiling

The quality of the answers is capped by the quality of what you feed it, more than by the model. Three things matter most:

One topic per page. Retrieval works by finding relevant chunks. A single 4,000-word page covering shipping, returns, warranty and sizing retrieves badly for all four. Split it.

No contradictions. If an old page says 14-day returns and a new one says 30, the bot will sometimes say 14. Delete the old page rather than leaving it live and hoping. This is the single most common cause of a bot that “randomly gives wrong answers” — it isn’t random, it’s retrieving a stale document you forgot about.

Answer-shaped writing. Marketing copy retrieves badly. Write the question as a heading and answer it in the first two sentences. This helps your search rankings too, so it’s not wasted effort either way — and it’s increasingly how AI search engines pick what to quote.

Also: PDFs full of tables and scanned documents are much harder to work with than clean web pages. If your knowledge lives in a scanned handbook, expect that to be a real chunk of the project.

Guardrails: the part most projects skip

A chatbot on your website is a text box that anyone on the internet can type into, wired to a system that follows instructions. That deserves a moment’s thought before launch.

People will try to misuse it. Some will ask it to ignore its instructions, write their homework, or say something quotable about a competitor. Screenshots travel. The mitigations are unglamorous and effective: constrain it to your topic explicitly, have it decline anything off-topic, and never put anything in its instructions you wouldn’t want read aloud — treat the system prompt as public, because a determined user can usually get it out.

Be careful what you connect it to. A bot that can look up an order is useful. A bot that can cancel one needs identity checks first, and a bot that can issue a refund needs a human in the loop. Read access is a different risk category from write access — keep that distinction deliberate rather than accidental.

Watch what goes into the conversation. Customers will paste order numbers, addresses, sometimes card details they shouldn’t. Decide upfront how long transcripts are kept, who can read them, and whether they’re sent to a third-party model provider. If you operate under GDPR this needs to be in your privacy notice, along with the fact that an automated system is handling the conversation. It also needs to be true — a retention policy nobody implemented is worse than none.

How to know if it’s working

Don’t judge it on vibes after five questions. Do this instead:

Pull 50 real questions from your actual inbox, including the awkwardly worded ones and the ones with typos. Run all 50. Score each answer: correct, incomplete, wrong, or correctly escalated to a human. Wrong is the number that matters — an incomplete answer is annoying, a confidently wrong one costs trust.

Keep that set of 50. It becomes your regression test: every time you change the instructions, add content or switch model, run it again and compare. Without a fixed test set you’re making changes on impressions, and it’s genuinely easy to fix one answer while breaking three others.

Then track containment: the share of conversations resolved without a human. If a chatbot handles 60–70% of your repeat questions and cleanly hands off the rest, that’s a strong result. Chasing 95% usually costs more in build and maintenance than just having a person handle the last 30%.

One warning about containment as a metric: it goes up when customers give up. Read a sample of contained conversations each week rather than trusting the number alone, and watch for people asking the same thing three times before leaving.

Design the handoff before you design the bot

Most chatbot complaints are not about wrong answers. They’re about being trapped — the customer knows the bot can’t help and can’t find the exit.

Put a visible route to a human on every message, not buried after three failed attempts. Escalate automatically when the customer asks twice, when they sound frustrated, or when the retrieval step finds nothing relevant — a bot that answers from nothing is a bot that’s guessing. And pass the transcript with the handoff: asking someone to repeat what they just typed is the fastest way to undo any goodwill the bot earned.

When not to build one

I’d talk you out of it in three situations. If your support volume is genuinely low — a handful of emails a week — the maintenance will cost more than the time saved, and a good FAQ page does the same job. If almost every question needs live account data, you’re really buying an integration project, and the chatbot is the small part. And if your documentation doesn’t exist, fix that first: you’ll get value from it immediately in search and in onboarding, whether or not a bot ever reads it.

The setup that actually works

Ground it in your content. Tell it explicitly to say “I don’t know, let me connect you with someone” rather than guess. Show the source page under each answer, so customers can verify and your team can debug. Give it a visible escape hatch to a human on every message. Review the transcripts weekly for the first month, because that’s where your next ten knowledge-base pages are hiding.

A bot that admits its limits and hands over cleanly gets trusted. One that bluffs gets switched off within a month.

If you want one built properly — grounded, guarded and measured — see AI chatbot development. If you’re not yet sure whether a chatbot is even the right thing to build, AI consulting starts from the business problem instead, and the five-day audit is a good way to find out on your own first.