Chatbot audits
Does your chatbot know your own policies?
Every wrong answer your chatbot gives can cost you a sale, a refund or a customer. I test it against your own policies and show you where it could be losing you money.
Book a 15-minute callor call 07472 114734
Uses only your public website. Nothing for you to set up.
- Assistant: Hi, I am Ellie, the Example Retail assistant. How can I help?
- Customer: What is the return window for an item bought online, and are there exclusions?
- Assistant: You can return items within 30 days of delivery for a full refund, as long as they are unused. Personalised items cannot be returned. Returns policy
Doesn't match the policy
The page it should have matched
Returns policy
You have 28 days from delivery to return an item for a full refund. Items must be unused and in their original packaging. Personalised items cannot be returned unless they arrive faulty.
- The real window
- Left out of the reply
What I test12 real customer scenarios
What I checkEvery answer, against your own pages
What you getA signed-off report in 5 working days
Why does it matter?
Your chatbot could be losing
you money right now
Your chatbot speaks for your business, all day, to customers who take it at its word. When it gets something wrong, you pay for it.
“…you are responsible for what an AI agent does in the same way you are responsible for what an employee does.”
-
01
A wrong refund window
The assistant says: You can return items within 30 days of delivery.
Your policy says 28.
What it costsRefunds you didn't owe, or disputed payments that cost you the sale and a fee.
-
02
A fee that doesn't exist
The assistant says: There is a £4.99 restocking fee on all returns.
Your policy lists no fee.
What it costsAbandoned orders: customers walk away before they pay.
-
03
No way to reach a person
The assistant says: Sorry, I can't help with that. Is there anything else?
No phone, email or contact form offered.
What it costsBuyers with one last question go cold and spend their money with a competitor.
-
04
Different answers on different pages
Home page
The assistant says: Returns within 30 days.
Help page
The assistant says: Returns within 28 days.
What it costsYou pay twice: for the chatbot, and for the staff answering what it got wrong.
Fictional examples.
Sales at stake each month
1,000 chats × 1% × £60 =
£600
£7,200 a year
The audit: £500, once.
Your numbers, your arithmetic
A what-if, not a forecast or a claim about your chatbot. It only counts sales lost outright. Refunds, dispute fees and staff time come on top.
What is it, exactly?
A mystery shopper for
your chat assistant
Your assistant answers customers all day. I check what it actually tells them: the questions your customers really ask, put to it the way they'd ask them, with every reply checked against the page that governs it.
-
01
Policy accuracy
Wrong refund, delivery or warranty terms, quoted as fact.
-
02
Uncertainty and sources
Made-up answers where it should say it doesn't know.
-
03
Human handoff
Customers who can't reach a person when they need one.
-
04
Cross-page consistency
A different answer depending on the page it's asked on.
-
05
Tone and brand fit
Off-brand, off-topic or careless with an upset customer.
-
06
Speed and recovery
Slow replies, and dead ends when it doesn't understand.
Each area is scored on a fixed scale, with the evidence behind every score.
How does it work?
Three stages, and nothing
for you to set up
It uses only your public website, as an ordinary visitor would.
-
1
Capture
Your published policies, saved and fingerprinted on the day, so every finding traces back to your exact wording.
-
2
Test
Structured customer scenarios across six risk areas, with every reply logged and timestamped.
-
3
Verify
Every failure is re-tested in a fresh session before it's reported. Then you get a scored, signed-off report.
What I never do
- Log in, or use real customer details
- Open tickets or ask for a human agent
- Run up your chatbot bill
- Publish or share your results without your written OK
Can't we test it ourselves?
Your chat tool lets your team try questions. What it won't give you:
- An outside check. I don't build, sell or fix chatbots, so I've no reason to mark it kindly.
- Your own policies as the yardstick. Every answer is checked against the page that governs it, not against what sounds right.
- Evidence that holds up. Timestamped replies and fingerprinted copies of your pages, so anyone can check a finding later.
What do I get?
Every problem,
beside your own policy,
with a suggested fix
A signed-off report your team can work straight from. Here's a fictional sample, so you can see exactly what you'd get.
- A score for each of six areas, tied to the questions behind it
- Every question, with the assistant's exact reply and the time
- Each problem beside your own policy, with what it costs a customer and a suggested fix
- Suggested fixes, ranked by impact, then effort
- Questions for your team, kept apart from the findings
- The full evidence record (transcripts, saved pages and their fingerprints), yours on request
How much is it?
£500 for the audit
One price for the 12 scenarios, the re-tests and the signed-off report.
How do I start?
Find out what your chatbot
is telling your customers
I'm Archie George, based in Essex. I run every audit myself, from the first question to the signed-off report. Book a call and I'll take it from there.
or call 07472 114734