The 20-Point AI Bot Audit We Run Before Touching Anything

You would not let a mechanic replace your transmission on a hunch, before running a diagnostic. Yet that's how most AI bots get "fixed": a symptom appears, somebody guesses a cause, a freelancer rewrites the prompt, and three weeks later the same failure returns wearing different clothes. Fixing without auditing is how you pay twice. So we standardized the inspection. Before we touch any client's bot — before we even quote — we run the same 20-point audit, every time. This article is the entire checklist, published. Run it yourself, hand it to your developer, or have us run it. The order of operations matters more than who executes it.

Why audit before fixing

Because symptoms are shared across causes. A bot that gives wrong answers might have any of five distinct diseases — we mapped them in Why Your AI Chatbot Gives Wrong Answers — and an agent that misfires has seven of its own. Treating the wrong layer wastes the budget and often adds regressions, because every untracked change to a working system is a new risk. The audit produces the thing repairs should start from: evidence. Twenty graded points, five layers, and a ranked list of what's actually broken versus what merely looks suspicious.

The 20 points

Data layer — what the bot knows

1. Knowledge-base freshness. Every source has a last-updated and last-indexed date, and the two are close together.
2. Coverage map. The top real customer questions each trace to a source document; the ones that trace to nothing are listed.
3. Chunking quality. Retrieved passages keep tables, prices and policy clauses intact instead of slicing them mid-thought.
4. Retrieval hit rate. Replaying real logged questions returns the right document near the top, measurably, not anecdotally.

Prompt layer — what the bot is told

1. Prompt inventory. One canonical system prompt exists, version-controlled, with a named owner — not four variants in four tools.
2. Contradiction scan. No instruction conflicts with another; accumulated "quick tweaks" are reconciled or deleted.
3. Refusal and escalation rules. The bot has explicit permission to say it does not know, and a working handoff path to a human.
4. Scope constraints. What the bot may not discuss — legal, medical, competitors, pricing exceptions — is written, not assumed.

Tool layer — what the bot can do

1. Tool inventory and descriptions. Every connected tool has a distinct, unambiguous description; overlapping tools are merged or split.
2. Argument validation. Amounts, IDs, emails and enums are checked before execution, so a mistyped refund cannot run.
3. Timeout and fallback behavior. When a backend is slow or down, the bot says so instead of answering without the data.
4. Credential health. Token expiry dates are known, rotation is scheduled, and scopes follow least privilege.
5. Idempotency on destructive actions. Anything touching money, messaging or records carries a unique key so retries cannot double-fire.

Monitoring layer — whether anyone would notice

1. Action and conversation logging. Every reply and tool call is recorded somewhere a human can actually read in minutes.
2. Heartbeat alerts. An alarm fires when an expected run does not happen — the only cheap defense against silent death.
3. Loud errors. Failures surface to a channel someone watches; nothing is swallowed into a log nobody opens.
4. Regression test suite. A golden set of known-hard questions runs before every prompt or data change ships.

Cost layer — whether the meter is guarded

1. Budget caps. Hard per-run and per-day limits on tokens, spend and action counts exist at the platform level.
2. Context growth. Token usage per conversation is tracked over time and is not quietly climbing month over month.
3. Model-to-task fit. The expensive model is not doing work a cheap one handles — and the cheap one is not faking work that needs the strong one.

The audit at a glance

How we score it

Every point gets pass, partial or fail, weighted by blast radius — a failed point 13 on a refund-issuing agent outranks a partial point 8 on a FAQ bot. The deliverable is a written scorecard, the top three risks in plain language, and a fix plan ordered by risk-per-effort: quick wins first, structural work second. The report is deliberately portable. Any competent developer can execute it, which keeps us honest — the audit has to stand on its findings, not on lock-in.

Can you run this yourself?

Partly, and genuinely. Points 1, 5, 14 and 18 are an afternoon of honest work: check the dates, find the canonical prompt, open the logs, set the caps. If you run only those four, you're ahead of most deployments we see. The hard points are 4 (retrieval replay needs your real query logs and tooling), 6 (contradiction scanning across a 2,000-word prompt is harder than it sounds), and 13 (idempotency review means reading how every destructive call handles a retry). That is where a paid audit earns its fee: fixed price, a few business days, and a findings report you keep either way — including when the finding is your bot is mostly fine, here are the three gaps.

What happens after the audit

Three outcomes, in descending frequency. Most bots need configuration-level fixes — re-indexing, prompt cleanup, caps and alerts — measured in days. Some need structural work: re-chunking the knowledge base, redesigning tool definitions, adding a test suite. A few need the honest bad news: this architecture will keep failing, rebuild the core. We quote fixes separately from the audit precisely so that recommendation stays trustworthy — an audit that always concludes hire us for a big rebuild is a sales funnel, not an inspection.

If your bot is misbehaving, or you simply don't know its grade, request the 20-point audit. Tell us what the bot does and what it is doing wrong, and we will confirm scope and turnaround within one business day.

Keep reading