Why Your AI Chatbot Gives Wrong Answers (and How We Diagnose It)

Your chatbot passed every test in the demo. Three months later it is quoting a return policy you retired in January, inventing a phone number for a support line you never had, and promising two-day shipping on items your warehouse takes ten days to move. None of this means AI is unreliable by nature, and none of it is random. A chatbot that gives wrong answers is a system with a fault in one of five specific places, and each fault leaves its own fingerprints. This is the taxonomy we use at botsfix when a client sends us a transcript and asks what went wrong — and the diagnosis table we walk through before we agree to change a single line.

Wrong answers are a symptom, not a diagnosis

Nine owners out of ten respond to a lying bot the same way: open the system prompt, add a line like never make up information, redeploy, and hope. Sometimes that buys a quiet week. It almost never holds, because the prompt is rarely the only thing that failed.

A wrong answer is the visible end of a chain: what data the bot can see, how that data is retrieved, what instructions shape the reply, how the model behaves at the edge of its knowledge, and whether the systems feeding it responded in time. A fix that doesn't identify which link broke is a coin flip. So the first question is never how do we fix it but which of the five failure classes does the evidence point to.

The five root causes

1. Stale knowledge base

The most common cause and the least glamorous: the bot is faithfully reporting yesterday's truth. The FAQ page was updated but the search index was never rebuilt. A PDF price list from two quarters ago is still sitting in the corpus. Product descriptions were synced once at launch and never again. The fingerprint is distinctive — the wrong answer was correct at some identifiable point in the past. If the bot's claim matches an old version of your website, you have a data freshness problem, not an AI problem. The fix is unglamorous too: a re-indexing pipeline, expiry dates on content, a sync schedule, and deleting the zombie documents nobody remembers uploading.

2. Bad chunking and broken retrieval

Most business bots use RAG — retrieval-augmented generation. The bot searches your document library and passes the best matches to the model, which answers from them. Two things go wrong. First, chunking: documents get sliced at arbitrary boundaries, so a price ends up in a different chunk than the product it belongs to, or a table is cut in half mid-row. Second, retrieval itself: a customer asks about cancellation and the search returns the marketing page that mentions the word, not the actual policy. The fingerprint is an answer that is plausible in shape but assembled from mismatched fragments — right template, wrong numbers — or a flat I don't have information on that when the document definitely exists. The fix approach is to replay real queries, inspect exactly which chunks came back, re-chunk along semantic boundaries, tune the retrieval settings, and add reranking where needed.

3. Prompt drift

Prompts accumulate. The launch prompt was 300 tidy words; a year of quick tweaks later it is 2,300 words of layered instructions that quietly contradict each other — be concise but always thorough, never mention competitors except when comparing plans, always offer the discount but only sometimes. Nobody remembers who added what, and nothing is version-controlled. Models handle contradictions unpredictably: they obey whichever instruction sits closest to the end, or blend both into something nobody wrote. The fingerprint is inconsistency — similar questions get different answers on different days, and regressions appear right after an unrelated small change. The fix is prompt archaeology: reconstruct the history, delete the contradictions, restore a single voice, then put the prompt under version control with a change log, like the production code it is.

4. Hallucination on edge cases

Genuine model hallucination is real, but it's overdiagnosed — and it concentrates at the edges: questions your data never answers, ambiguous phrasing, rare products, languages you never planned for. When retrieval returns nothing useful and the prompt gives no explicit permission to say I don't know, the model does what it was trained to do: produce the most plausible-sounding completion. Plausible is exactly the problem — invented tracking links, confident nonsense about warranty terms, a made-up support email. The fingerprint is fluency about things that do not exist. The fix approach: explicit refusal instructions with an escape hatch to a human, grounding rules that restrict answers to retrieved context, and an edge-case test suite so the same failure cannot return unnoticed.

5. Integration timeouts and silent fallbacks

A business bot sits in a chain: website widget, orchestration layer, model API, then your CRM, order system or calendar. When the order-status API takes nine seconds and the timeout is five, many stacks don't show an error — they quietly let the model answer without the data. The model, still obligated to be helpful, guesses. The fingerprint is clustering: wrong answers spike at peak hours, or only one intent misbehaves, or the same question is answered correctly on retry a minute later. That pattern is never the model. It's plumbing. The fix is to trace the full request path, log tool failures loudly, and make the bot say I can't reach the order system right now instead of improvising.

The diagnosis table

Rule of thumb: match the symptom before proposing the surgery.

How we actually run the diagnosis

The order matters, because each step narrows the next one.

1. Transcript sampling. Pull 50–200 recent conversations and tag every wrong answer against the five classes. The distribution alone usually names the primary suspect.
2. Retrieval replay. Re-run the failing questions against the knowledge base and read exactly which chunks came back, in what order.
3. Prompt archaeology. Reconstruct the prompt's edit history and flag contradictions, dead instructions and duplicated rules.
4. Edge-case suite. Fire a battery of known-hard questions — out-of-scope topics, ambiguous phrasing, adversarial follow-ups — and score the refusal behavior.
5. Integration trace. Follow one bad conversation through every hop with timestamps, looking for the quiet timeout.

Evidence comes before edits. By the end of this sequence we can say your bot is wrong because X, show the receipts, and estimate the fix — instead of billing you for educated guessing.

Why "just switch the model" rarely works

Upgrading the model is a new engine in a car with a clogged fuel line. A stronger model summarizes stale documents more eloquently — it's still wrong. It reasons beautifully over the wrong retrieved chunk. It waits politely on the same timeout. We do recommend upgrades, but after the pipeline is verified clean, and only when the residual failures are genuinely about reasoning. Most of the failures we see die in data, retrieval or integration long before model quality matters.

What to do next

If your bot is misbehaving, you have two honest options. Do it yourself: start with the table above, then read our teardown of agent-side failures in When Automations Go Rogue and the full inspection sequence in The 20-Point AI Bot Audit. Or have us do it: we run the complete diagnosis on your live bot, deliver a written findings report with severity rankings, and quote the fix separately so you can take it anywhere. Request an audit and we will tell you within one business day whether your case fits.

Keep reading