The AI Voice Agent Benchmark Playbook

What you'll get

Pricing pages lie. Every voice AI platform claims sub-second latency, human voices, and hallucination-proof agents. The only way to know is to run identical agents on real calls and score them. This playbook gives you:

• The exact 4-metric scoring system we used: latency, voice quality, real price, hallucination rate
• Copy-paste commands to deploy the same agent on Bland, Vapi, and Retell
• A real-cost worksheet that exposes what a minute actually costs on each platform
• A decision matrix so you pick based on your use case, not their marketing

Lock your variables before you test anything

A benchmark is worthless if the agents differ. We used one identical setup across all four platforms: same system prompt, same ElevenLabs voice, same LLM (GPT-4.1). Change one variable per platform and you're comparing prompts, not platforms.

Write one system prompt for one job. Ours was a sales qualification agent: greet, ask three qualifying questions, book or bail. Keep it under 300 words. Save it as prompt.txt and use that exact file everywhere.

You are a qualification agent for [company].
Ask exactly 3 questions: budget, timeline, decision maker.
If qualified, offer a booking link. If not, thank and end.
Never invent pricing, features, or availability.
If you don't know, say "let me have a human follow up."

That last line matters. Most "hallucinations" on real calls are the agent inventing answers instead of escalating. Give it an exit.

Deploy the identical agent on each platform

Bland is a straight API call. Task in, call out:

curl -X POST https://api.bland.ai/v1/calls \
  -H "authorization: $BLAND_KEY" \
  -H "Content-Type: application/json" \
  -d '{"phone_number":"+1XXXXXXXXXX","task":"<your prompt>"}'

Vapi is dev-first. You bring your own stack, which is the point and the trap:

vapi assistant create --name sales-qual
# stack: Deepgram STT + GPT-4.1 + ElevenLabs TTS

Retell works through its dashboard or API. Pull call logs later with:

curl -s https://api.retellai.com/v2/list-calls \
  -H "Authorization: Bearer $RETELL_KEY"

Synthflow is the no-code option: build the agent in their visual editor, paste the same prompt, select the same voice. No CLI needed, which is exactly who it's for.

Run 50 real calls per platform

Not demos. Real outbound calls to real numbers (your own team, opted-in leads, or test lines you control). We queued 200 calls through Twilio, 50 per platform. If 50 is too many for day one, run 10 each. The pattern shows up fast.

Log four things per call:

• Latency: time from when you stop talking to when the agent responds. Pull p50 and p95 from platform logs.
• Voice quality: does it read robotic on long replies? Does it handle interruptions or talk over you?
• Completion: did the call finish the job, or did the human hang up in under 30 seconds?
• Hallucinations: read the transcripts. Count every invented price, feature, or promise.

The 700ms rule: below 700ms response time, calls feel human. Above it, people start saying "hello? you there?" and drop.

Calculate the real cost per minute

Every platform advertises a base rate. Almost none of them are what you'll pay. Here's what our test surfaced:

The worksheet: platform fee + STT + LLM + TTS + telephony = real cost. Run that math before you commit. As a rule, budget 2 to 3x the headline price.

Pick by use case, not by hype

Then re-test quarterly. These platforms ship fast, and today's loser fixes its latency next quarter. Your benchmark harness is reusable; the rankings aren't permanent.

The checklist

• One prompt, one voice, one LLM, locked across every platform
• Hallucination escape hatch written into the prompt
• Same agent deployed on Bland, Vapi, Retell, Synthflow
• 10 to 50 real calls per platform, logged
• p50/p95 latency, completion rate, and hallucination count recorded
• Real cost per minute calculated with the full-stack worksheet
• Platform picked by your use case, calendar reminder set to re-test in 90 days