What three months of an AI voice agent taught us

For the last three months, an AI voice agent has been answering the phone at Manchester PC (opens in a new tab). We built it ourselves, from scratch, as the first real test of what will become part of Step Into AI's AI receptionist (opens in a new tab). This is what we learnt: the problems, what we fixed, what we have not, and whether it has been worth it.

Why we did it

Small businesses lose a surprising amount of work through missed calls (opens in a new tab). Calls arrive while you are with a customer, under a desk or halfway through a repair. You either stop what you are doing or the call goes unanswered. I wrote about that wider problem in what actually changes when the software starts answering.

So we put an AI voice agent on our own phone line. Testing it on our own business first meant every mistake cost us, not a client.

Getting the basics right was the hardest part

Holding a conversation turned out to be the easy bit. Capturing basic details accurately was not. Postcodes, names and email addresses caused the most trouble.

We tried almost every speech-to-text model we could find. Some were better than others, but none were consistently reliable at these three things.

Postcodes: check against the real thing

It took several weeks of testing to spot the pattern. Most postcode errors happened in areas with two-digit districts, such as M20 or M50. The agent usually heard most of the postcode correctly, but not all of it.

The fix was to stop relying on the transcription alone. We added a postcode knowledge base, so the agent compares what it heard against real UK postcodes and works out the most likely match. Open services such as Postcodes.io (opens in a new tab) make UK postcode lookup and validation freely available. We do the same for names. It sounds obvious now. It was not at the time.

We only need the postcode at the moment. If we ever need the full address, the sensible approach is to take the postcode first and look the address up, so there is less for the phone line to mishear.

Email addresses: stop asking voice to do a text job

Email addresses were a different story, and we still have not found a voice-only solution we are happy with.

Instead, the agent asks its questions, records the answers and sends the caller a text message with a link to review their details and add their email address. The agent handles around 250 calls a month, and around 60% of the callers who get the text complete it. That is better than I expected.

When I shared this, one reply put it perfectly: the email problem was never a transcription problem, it was a channel problem. Spelling an address out loud fails between two people too. Call centres have read addresses back letter by letter for decades. The text message works because we stopped asking voice to do a text job.

Other suggestions I am considering:

• Ask for a reply instead of a form. “I will send you a text on this number, just reply with your email address.” One less step.
• Look the caller up first. If the number has called or written before, the agent already knows the name and only needs to confirm it.
• Confirm only the tricky part. Spell back the part before the @ letter by letter, then offer the common providers as a choice.

The extra step still bothers me. It is one more thing the caller has to do. But a wrong email address is worse than a short text.

Always tell callers it is AI

Our agent introduces itself as AI at the start of every call. That was never up for debate.

I have seen plenty of discussion about making AI agents sound so human that callers never realise. I disagree with that approach. People should know straight away who, or what, they are talking to. Trust is the whole business, and it is very hard to get back once a customer feels tricked.

Latency: the awkward silence

In the early days there were noticeable pauses while the agent processed what it had heard. We filled them with the sound of keyboard typing. It sounded like something from the 1980s.

We removed it and focused on making the work behind the scenes faster. It is much better now, though there is still room for improvement. Silence on a phone call feels much longer than the same pause in a text chat.

Background noise

People call from busy streets, cafés, cars and rooms full of other conversations. Early on, the agent would sometimes pick up stray words and try to make sense of them.

We added noise cancellation and changed how the agent treats information that does not make sense in the context of the conversation. Problems have become rare.

Two technical tips from people who have fought the same battle are worth passing on. First, phone audio is narrowband. Voice activity detection tuned on high-quality audio can miss more speech on a phone line, so check what yours was tuned on. Second, noise cancellation removes noise but keeps other voices, because they are speech too. Isolating the caller’s voice can help more than general denoising.

Do customers actually want to talk to AI?

This worried me more than anything before we started. In practice, most callers are perfectly happy, and some seem to enjoy it. People have asked the agent what platform it runs on and which models it uses. One caller tried to get a cake recipe out of it. In every case we have seen, it stayed on task and politely declined to go off-topic.

Some callers ask for a person straight away, and that is completely understandable. There must always be a quick route to a human.

Has it helped the business?

Massively. The biggest benefit is not the cost saving. It is no longer being interrupted all day by the phone.

We can get on with the work, then review the information collected and decide which enquiries need a call back and which can be handled by text or email. For a small business, that is a huge difference in how the day feels.

The downside: conversion fell from about 30% to 22%

There is one result I am not going to hide. Our conversion rate used to sit at around 30%. It now sits at around 22%.

I cannot say the voice agent caused it. Our leads and our conversion were already falling before the agent went in, so the decline started without it. This year has been different in other ways too. People may simply have less money to spend on repairs, and we have changed more than one thing at once, so I cannot separate the causes cleanly. Search in particular has changed. Pew Research Center (opens in a new tab) found that Google users who saw an AI summary clicked a traditional search result in 8% of visits, against 15% for those who did not. We are seeing that shift in how customers find us and in the kinds of enquiries we receive.

But I am not ignoring it either. There is no point saving money on answering calls if you lose more through missed sales. A good suggestion was to break the drop down by where people leave: during the call, at the text message step, or while waiting for a call back, and to compare similar enquiries handled by a person. That is what we are doing now.

If you want to understand your own numbers before trying any of this, keep a simple log for one normal working week: every call, message and form, the time it came in and the time someone first replied. Counting what you miss is free, and it is worth knowing before you pay for anything. If you are weighing up the options, I compared them in AI receptionist or human answering service.

Would we go back?

Probably not. The agent does its job well enough that it is hard to imagine returning to stopping work every time the phone rings. It is not perfect, and I understand why some customers would rather speak to a person.

I do not like the idea of replacing human contact with AI everywhere. But costs keep rising, and small businesses have to find ways to work more efficiently without lowering their standards. That is the reason I am building an AI platform for service businesses, with a person always in control of what the system does.

My biggest lesson is this. Building an AI voice agent that can hold a conversation is relatively easy now. Building one that reliably handles real customers, noisy streets, unusual requests and all the small things that happen on a real phone call is a completely different challenge. It is the same gap I described in what AI agents can actually do for small businesses: the demonstration is the easy part.

We are still learning, and I expect to be improving it for a long time.