AI for custom customer care: what it can answer and what it must not

The question that actually decides the project

Almost every AI customer care conversation starts in the wrong place. It starts with which model, which platform, which vendor — and those are the least consequential decisions available. The models are broadly capable, the platforms are broadly similar, and a project that is scoped correctly will work on several of them.

The consequential question is narrower and harder: which of your customers' questions are you willing to be wrong about? Not in the sense of accepting failure, but in the sense of designing for it. Every automated answer has a cost when it is wrong, and those costs are wildly unequal. Telling someone the wrong opening hours is an inconvenience. Telling them the wrong dosage, the wrong entitlement, the wrong contractual term or the wrong refund policy is a different category of event entirely.

Scope the system by that cost and it usually works. Scope it by what the technology can plausibly attempt and it usually does not, because the failure modes land in exactly the places where a wrong answer is expensive.

Sorting questions by the cost of being wrong

The practical exercise is to take a month of real contacts and sort them, not by volume or by topic, but by what happens if the answer is confidently wrong. Three tiers emerge almost every time.

The first tier is informational and low-consequence: hours, locations, order status, how to reset something, where to find a document. A wrong answer here is annoying and recoverable, and the customer usually detects it immediately. This tier is frequently a large share of contact volume and it is genuinely well suited to automation.

The second tier is transactional and moderately consequential: booking changes, address updates, returns within a stated policy. These can be automated but they need to act on real system state rather than describe it, which makes them integration projects rather than content projects.

The third tier is consequential and contested: anything touching money owed, legal position, health, safety, eligibility, or a customer who is already unhappy. The right design here is not a better prompt. It is a fast, unconditional route to a person, and a system that recognises it should not be answering.

Tier one — informational

Wrong answers are annoying and self-correcting. Automate freely and measure containment.

Tier two — transactional

Wrong answers cost time and goodwill. Automate against live system state, never against a description of it.

Tier three — consequential

Wrong answers cost money, trust or safety. Detect and route to a person immediately, without a negotiation loop.

Grounding matters more than the model

A general-purpose model asked about your business will produce something plausible, because producing something plausible is what it does. That is the entire problem. The fix is not a larger model or a longer instruction; it is retrieval — the system finds the passage in your own documentation that answers the question, and the model composes an answer from that passage rather than from what it happens to remember.

This has a second benefit that is often more valuable than the accuracy: the answer can cite where it came from. A customer care answer that says which policy document and which section it is drawn from is checkable, and a checkable answer is one your own staff can trust when it is escalated to them.

It also relocates the hard work, honestly. If the answer must come from your documentation, then your documentation is the product. Most organisations discover during this project that their policies exist in three versions, that two of them contradict each other, and that the authoritative one is in an email. Resolving that is unglamorous and it is the majority of the effort.

Refusal is a feature, not a failure

The single most useful behaviour to design is the refusal. A system that says "I do not have a reliable answer to that, let me put you through" is doing exactly what it should, and the rate at which it does so is a metric worth watching upward as well as downward.

This is harder than it sounds, because language models are not naturally calibrated about their own uncertainty. The practical mechanisms are structural rather than conversational: if retrieval returns nothing above a relevance threshold, refuse. If the question matches a defined sensitive category, refuse before generating anything. If the customer has expressed dissatisfaction, hand over regardless of whether an answer is available.

Teams frequently resist this because containment rate — the share of contacts resolved without a person — is the number a business case was built on. But containment achieved by answering tier-three questions is not a saving; it is a liability being recognised later and larger.

Design the handover before the answers

The handover is where most deployments are actually judged. A customer who has spent four minutes with an automated system and is then dropped into a queue at position twelve, repeating everything they have already typed, has had a worse experience than if the system had never existed.

A competent handover carries the full transcript, whatever the system already retrieved, and any identity or account context established, into the agent's view before they speak. It should also be reachable at any point, without a keyword the customer has to guess, and without the system attempting to talk them out of it.

The measure worth tracking is not how many contacts were contained but how many were resolved end to end without the customer repeating themselves. That number is harder to move and it is the one that correlates with whether people will use the channel again.

Where the regulatory line sits

Two things need establishing before build rather than after. The first is transparency: under the EU AI Act, systems interacting with people generally have to make clear that the person is dealing with an AI system, and in practice this is straightforward — say so plainly at the start rather than in a footer.

The second is decisions. If the system determines eligibility, entitlement, creditworthiness, access to a service or anything else producing a legal or similarly significant effect, you are in a different regime — GDPR Article 22 on automated decision-making, and potentially high-risk classification under the AI Act. Customer care systems drift into this territory more easily than people expect: an assistant that tells a customer whether their claim is covered has made a determination, whatever the interface calls it.

This is not legal advice and the position depends on your sector and jurisdiction. It is a scoping fact: settle it with your advisers before the build, because retrofitting human oversight and documentation into a shipped system is considerably more expensive than designing for it.

What a realistic first phase looks like

A first phase that works tends to look boring. One channel, tier-one questions only, grounded in a defined and cleaned set of documents, with an immediate handover and a measured baseline for comparison. Four to eight weeks is a realistic range for that, with a meaningful share of it spent on the documentation rather than the software.

The baseline matters more than most teams allow for. Contact volume by type, current handling time, current resolution rate and current satisfaction, measured before anything is deployed, is what turns the second phase from an argument into an arithmetic exercise. Without it, every subsequent discussion about expansion is a matter of opinion.

Expansion then follows the evidence: the categories where containment is high and correction rate is low move into scope, and the categories where the system keeps being wrong stay out until the underlying documentation is fixed. That is a slower path than most vendors describe, and it is the one that survives contact with real customers.

The failure patterns worth recognising early

Three patterns account for most of the disappointing deployments we see. The first is scope set by volume: automating the largest contact category regardless of consequence, because that is where the apparent saving is. This produces a system that is confidently wrong about the things that matter most.

The second is documentation left as it was. If the retrieval corpus contains contradictions, the system will answer with whichever contradiction it retrieved, and it will do so fluently. Teams then conclude the technology is unreliable, when what is unreliable is the source.

The third is measuring containment alone. It is the easiest number to move and the easiest to move in the wrong direction, and a deployment optimised for it will systematically avoid the handover that a proportion of customers actually needed.

Where this sits in what we do

This article covers one decision inside a wider engagement. The solution page sets out how that engagement runs, what it includes and what it costs to find out.

• AI Customer Experience — A chatbot, a voice agent and a knowledge base that share one grounded source of truth — with citations, measured accuracy and a fast route to a human.
• When to build software instead of buying it
• Why more traffic rarely fixes a pipeline problem
• Lead scoring that sales will actually use
• Quote to cash without re-keying: where the handoffs break
• All insight articles