I spent 204 minutes calling AI answering services. Here’s what surprised me.

Over ten days in October and November, I made 103 calls. 96 calls were to working home service companies (HVAC, plumbing, electrical, pest, garage doors, junk removal, roofing) and 7 calls were to vendors’ public demo lines.

Of the 103 calls, 49 were answered by a human, 14 went to voicemail or rang out, 6 died in a phone menu, and 34 reached an AI agent.

These were real calls to actual home service companies in the US, across 31 states. I called the same lines that their customers were calling that week. So while this might not be a huge sample size, the learnings are not simulated. This is the real deal.

I asked the same 10 questions in almost every call, posing as a homeowner with a problem—a busted heater, flickering lights, or mice in the kitchen. A few standard questions, like:

• “Do you cover my zip?”
• “What’s the soonest you can come?”
• “What’s the service fee?”

And then a few curveballs:

• “Sorry, my dog’s barking,” with ten seconds of silence.
• “Are you a real person or a robot?”
• “What’s 17 times 23?”

If a human answered or I got transferred to a real person, then I ended the call. I wasn’t evaluating human schedulers, and I didn’t want AI booking a fake slot for a real job and then missing out on revenue. (That would have been cruel.)

Every AI call was transcribed and scored on seven items, from whether it understood the problem to whether it could get me in front of a real person.

Honestly, I was pretty impressed by most of the AI agents I talked to. They would have fooled my grandma, and probably would have taken care of what she needed.

With a simple enough request, most of these agents would have gotten me scheduled, told me the price, and sounded pleasant doing it. Here’s what they did well:

When I asked about price, 67% of AI answerers gave me a dollar figure. The rest declined or deflected, often for the right reason. One garage door company’s AI said: “I can’t quote repair prices or set up financing myself.”

85% of AI agents told me directly that they were not human. Most used soft language like “digital assistant” or “virtual care specialist.” One HVAC company artfully gave me options at the start of the call: “If you would like to speak with Regina, our automated agent, press one. To speak with a live customer service representative, press two.”

Only 15% dodged the question entirely, sometimes immediately redirecting me to a person. One pest company stated up front: “This is an AI agent. However, I can take a message, start, stop, or even reschedule an appointment if needed.”

After letting me know that no one could come out tonight, a plumbing AI suggested an alternate fix: “In the meantime, if it’s seeping, you can turn the shutoff valve behind the toilet clockwise to stop the water.” An electrical AI asked, before anything else, whether there was fire, smoke, or immediate danger. Another offered to text me the financing information while I was still on the line.

About halfway through each call, I’d say, “Just a sec, my dog’s barking.” I wanted to see if the AI could understand that I needed a moment and respond with empathy.

Every AI agent I tested this interruption on was able to react, though they all had a delay. (This is one easy way to tell you’re talking to AI—humans have better response time). Most were thoughtful: “No problem. Take your time.” One AI even made a joke: “Dogs always have perfect timing.”

Only two AI answerers had an inappropriate reaction—trying to upsell me during the interruption. “Take your time. While you’re settling the dog, I’ll just add this.” No break or wait time, just plowing through the pitch.

During one call session, I was greeted by Tucker. His audio was poor, and he had a slight Southern accent, so I wrote him off as a real human and quickly ended the call.

However, half an hour later, I called a different company with the same greeting. Tucker sounded familiar, but the background noise sounded artificial, so I continued the call. Turns out that Tucker was a bot (likely from Avoca), and he was pretty convincing too. I called back the earlier line to verify, and wouldn’t you know? He turned out to be AI too.

I checked recordings from other calls that I had hung up on, thinking they were real people. Three more calls I’d written off as human were likely AI, once I compared the recordings.

While I was generally impressed with the AI answering services on the market today, there were some real misses. But it wasn’t one vendor that was notably worse than the others. It seems that the issue is how individual companies are setting it up, not how good the agent itself could be.

Here’s where I saw room for improvement:

We have customers who use AI for their after-hours line. My hit rate was notably better for bots when I called in the evening vs. during daytime hours—suggesting this practice is common across the board.

This can work well as long as you disclose the practice up front. For example, one HVAC company started with this line: “Our offices are currently closed. However, we do have a virtual agent who can help you. Please press 1 to speak with Emma, or 2 to leave a voicemail.”

The upfront transparency helps customers know what to expect. Unfortunately, about 12% of AI answering services implied that a human line was available (even after hours) and then couldn’t deliver. One bot opened the call with “If you’d rather speak with a live person, I can connect you.” Four minutes later, when I took it up on the offer, it said, “I can’t transfer you directly, but I’ll make sure your request goes to the team.”

Another announced, “One moment while I transfer you,” paused, and then: “I can’t transfer the call at the moment.” It did that twice.

When I mentioned emergency situations on a call (like no heat with kids in the house, a car stuck in the garage), about 15% of the AI agents were unable to pivot to an appropriate next step. An electric company promised to transfer me to a human, and then the call failed. An HVAC company tried to transfer, then said, “I can’t transfer,” but promised to “take notes.”

Three put me on an “urgency list.” One offered an emergency slot for $149 instead of $89, which is at least an honest answer.

If you’re old enough to remember the days of middle school phone tag, you’ll remember the goodbye loops that went on far too long. I got to relive those memories with a few committed AI agents. The longest loop lasted 7 minutes and 2 seconds.

“Are you not allowed to hang up on me?” I asked.

“I actually stay on the line until you hang up,” she/it replied.

“I give you permission to hang up. Goodbye,” I repeated.

“Thank you. All right. Thanks for the permission. Have a good night. Goodbye.” She did not hang up.

The bots that got stuck in a loop likely came from three different vendors—so it’s not a unique problem.

Other loops and glitches:

• An AI that asked me for my name eight times, even after I’d answered.
• Two different bots that quoted different prices for the same visit (“actually it’s $X”).
• One bot that introduced itself as Tina and later called itself Emily.
• A delightfully confused agent that switched between American and British English, and threw in an Australian phrase—just for fun.

Of the 34 AI conversations I had, a few vendors stood out.

Every AI conversation was graded on seven items, 1 to 5 each, for a total out of 35:

1. Understood the problem. Did it understand what I was calling about?
2. Intake. Did it ask for my details once, or loop?
3. Routing. Did it offer a real appointment slot, or honestly say when the next one was?
4. Urgency. Did it treat a heater out with kids at home like an emergency?
5. Handoff. When I asked for a person, did I get one?
6. Tone. Did it sound like someone I’d want answering my phone?
7. Stayed in bounds. No invented facts, no price flips, no transfers it couldn’t make.

A 35 is a CSR I’d hire tomorrow, a 25 is fine, and under 20 is a call you’d apologize for. Here’s how the top vendors compared.

The caveats? First, the vendors are our best guess: we matched them from each company’s public website, and only two AIs confirmed who built them. Second, the samples are tiny (as few as two calls per vendor), so a one-point gap is noise. Third, and most important: a home service company configures its own AI. The same product can be set to transfer or not, to say “AI” or not, to end a call or not. A low score here might mean the vendor shipped a bad default. It might also mean the business accepted one.

If I were a home service operator, I’d use this table to consider which vendors are actually in use today and how I might set them up to be more successful. (My hunch is, they’re all pretty much the same. So pick one that fits your needs and really invest in the setup. The better you train your bot up front, the less time you’ll spend rescuing customers from AI loops.)

Disclosure: I did use AI to help create my rubric and to score my call recordings. Do I suspect that Claude doesn’t have a perfect sense of what a warm and friendly tone may be? Yes, I do. Robots scoring robots feels apocalyptic, but this is the world we live in. I spot-checked scores to make sure they aligned with human taste, but these ratings are just one form of evaluation. Call a few vendor lines and see what suits your business.

Earlier this year, we analyzed 10,938 reviews of home service companies to see how customers feel about AI. Of the 125 reviews that mentioned AI, 78% were negative. Many of today’s customers still hate interacting with AI on a customer service call.

The most damaging pattern that we found in the reviews was AI booking an appointment the company couldn’t keep. “The AI booked me for a time slot they didn’t have, and I didn’t have an appointment,” one HVAC customer wrote. “An answering machine would have been better customer service.” That’s the one thing I didn’t test in this experiment, so the reviews complete the picture here.

The flip side is true too: Every positive review of AI that we found described an after-hours booking that a human honored the next morning.

The angriest reviews share one feature: the customer was in an emergency, and the AI controlled the only path to a person. “There is no emergency line to call because it is AI. We are desperate.” I hit a version of that wall 12 times in my experiment.

Reviewers also complained about AIs “asking for the same information three or four times before reaching a human who asks for it all again.” On one call, I was asked for my full name eight times.

Using AI answering services is a serious investment—and the cost is only part of that. Based on the variability I saw across bots built on the same platform, implementation and quality assurance (QA) matter if you want your customers to have a good experience.

For example, Avoca was the tool I reached most often, six times. Those six calls scored anywhere from 17 to 30. That’s the whole spread of the study, inside one product.

The difference between these Avoca calls was not the vendor, but the way the AI agent was set up.

It wasn’t just Avoca. Netic ranged from 19 to 30, and ScheduleBot from 21 to 31. My hunch is that you can reduce the variability in quality with intentional implementation and regular QA.

After this experiment, I’m actually more pro-AI than ever. The answering services worked really well for routine asks, leaving human agents free to handle the kind of escalations that turn a customer into an evangelist (or a detractor). Our own clients use AI in conjunction with our talent, and after 204 minutes, I understand why.

But if I were purchasing or updating my AI answering service today, I would not just set it and forget it. Here are a few things I’d train my agent to do:

One last consideration: If I were evaluating an AI answering service, I’d look closely at the invoice. It’s true that AI will never call in sick and can answer your phones 24/7. But does it cost you more than an FTE? And is the ROI really there? Check the math before falling prey to the promise of “cheaper” AI.

There was one breath of fresh air in my dive into the AI oceans, and it might surprise you.

On one call, a woman with a light Hispanic accent answered. I caught myself smiling. I was so used to talking to perfectly preppy bots with regionless, Midwest-adjacent accents that hearing a real human voice with a real accent was so refreshing.

If you want to delight customers with empathetic human answerers who also know their way around AI, reach out and we’ll set up some interviews. You might love them so much that you’ll be the one saying, “No, you hang up first.”