The hardest part of a voice agent isn't the AI. It's the silence.

The first real test call went like this. I phoned my own agent and, in Spanish, said I wanted to start a laundromat. Then I paused. Then I said I was looking for something to rent. Then I paused again and added two words that changed the whole question.

That small scene contains the whole project. A phone agent is a pipeline, and the pipeline is the easy part. The hard part is deciding, in the dead air between two sentences, whether the human is finished. Get it wrong one way and the agent interrupts. Get it wrong the other way and callers sit in silence, say "hello?", and start to wonder whether anyone is there.

What the client asked for, and why earlier attempts fell short

A Miami commercial real-estate brokerage wanted its main number answered by something that knew its listings and could hand callers to the agent who specializes in that area or kind of property. My partner Elizabeth, who was running marketing there and directed the project, told me they had already tried. Those systems sounded great on the phone and were rigid: they matched what a caller said against a prepared script, and the moment someone described a need that wasn't in the script, they had nothing to offer. A caller who says "something near Wynwood, maybe a warehouse, maybe to buy" is not on anyone's menu.

I had never built a voice agent. I was sure the answer was a language model, so I took it on.

The pipeline is the easy part

Twilio carries the call and streams the caller's audio. A speech-to-text service turns it into words. Claude decides what to say, and can call tools to search listings or log a lead. A text-to-speech service turns the reply back into audio. Each hop adds delay, and phone audio is poor to begin with: 8 kHz μ-law, the quality of a landline.

My first version used plain request and response, and it worked, but it felt like a walkie-talkie. Within days I rebuilt it as a streaming service over WebSockets, so audio flows in both directions all the time. Like any phone menu, it made callers press 2 for Spanish, and it spoke with Deepgram's built-in voices.

Better. Still clunky. And now the real problem showed up.

Problem one: when is the caller done?

Speech services mark a transcript "final" after a short pause, which is not the same as the caller finishing. People pause to think, to remember an address, to read a number off a screen. My first fix was a timer: collect fragments, and reply only after about a second and a half of silence. Next I used the service's utterance-end event, then tightened its settings: endpointing from 800 to 600 milliseconds, the utterance-end window from 3 seconds to 1.8.

Every one of those numbers is a trade. Wait longer and you never interrupt, but callers sit through dead air. Wait less and the agent is quick, and rude.

Every safety margin I added to avoid interrupting people was time the caller spent listening to nothing.

Problem two: my own judgment layer, and the answer it threw away

Background noise made it worse. A cough, a TV, a car passing could trip the agent into answering nothing. So I wrote a layer of rules I called the speech analyzer. For every utterance it weighed the speech service's confidence, the length, whether it sounded like a known intent, and then decided: respond, wait, or discard. A one-word utterance needed higher confidence than a sentence, and words from the business's vocabulary (neighborhoods, "warehouse", "NNN") got a lower bar, because "Miami" kept arriving as "ma'am."

It was sensible. It also silently ate the most important answer in the call.

Short answers are where phone speech recognition is weakest, and in a sales call, the shortest answers are the ones that carry the intent: buy, lease, yes, no.

Problem three: the phone mishears, in predictable ways

Once I started reading logs, the mistakes weren't random. Phone audio made the same confusions over and over, so I kept a table of them, and boosted the business's own vocabulary in the speech model so it was likelier to get the real word in the first place. A table of known mishearings is dangerous if it's broad: it will "correct" things people actually said. So the buy-or-lease fixes only apply at the moment the agent has just asked that exact question. In that moment a caller cannot be saying goodbye, so "bye" can safely be read as "buy."

Mishearings I saw repeatedly in test calls, and how the agent treats them.

I also tried to detect what callers wanted with broad pattern matching, and later removed most of it. It kept matching things people had never said. The overrides that remain are deliberately narrow, tied to one open question. Everywhere else, I let the model decide what someone meant, and use code for facts.

Voice, and a detour through a hosted platform

The fast built-in voices got me started. They were not the voice I wanted a business's name attached to. I switched to ElevenLabs' low-latency model, streamed in the format the phone network uses, with character-level timing, which I'll come back to. And rather than make people press 2, the agent now detects the language per turn and switches voices, in either direction, mid-call.

For a few days in March I also tried the managed route. A hosted voice-agent platform handles the hearing, speaking and turn-taking for you, and your server only supplies the words. My plan was to hand Spanish callers to a second, Spanish-speaking agent. The hand-off never took effect: the call kept going back to the English agent. In the logs I could see my server send the transfer and the platform ignore it.

That experiment clarified what I cared about. Hosted platforms are built to get you live quickly and, understandably, to keep latency low. To my ear, that came with compromises in how the voice sounded, and with decisions I couldn't see into or tune. This brokerage's callers were deciding whether to trust a company on the first ten seconds of a call. I wanted control of the voice, the prompt, the tools and the data, so I went back to my own stack.

Letting a model decide when the caller is done

In mid-March I switched to Deepgram's Flux, a speech model built for voice agents that does turn-taking itself, and my hand-built timing layers largely became unnecessary. Instead of final transcripts and silence timers, it emits turn events: a turn started, a turn probably ended, the speaker resumed after all, and a turn definitely ended.

The "probably ended" event is the interesting one. I configured it to fire at 50 percent confidence, with the definitive end at 70 percent. The moment the agent hears "probably", it starts drafting a reply with Claude, speculatively. If the caller resumes, that draft is cancelled. If the turn really ends, the draft is already underway. The cancel path matters: "Yes" in answer to a yes-or-no question usually means a longer sentence is coming, so short, low-confidence endings are held instead of answered.

Fixing hearing wasn't the same as fixing speaking. When a caller cuts in, the agent has to know what the caller actually heard before it stopped, not what it intended to say, because the model's next answer depends on it. I used Twilio's playback marks and the text-to-speech service's character timing to cut the agent's turn at the last word actually heard, and to rewrite the conversation history to match. That mechanism is the library this post is about.

Not every sound from the caller is an interruption. People say "uh-huh" and "sí, claro" while you talk, and an agent that stops for every one sounds jumpy. The library classifies short acknowledgements separately from a real utterance, so the agent keeps going.

Knowing the business

A receptionist that can't answer questions about the listings is just a polite menu. I connected the agent to the brokerage's Salesforce and kept the listings in memory, refreshed on a schedule, so a new listing becomes something the agent can discuss soon after it's added. Searching needed more than keywords: callers say neighborhoods, highway nicknames and half-addresses.

So I hand-built a small map of South Florida: neighborhoods, the highways everyone calls by nicknames, and how far apart things are. When nothing matches, the agent can say what is nearby instead of shrugging.

Most of the later fixes were prompts, not code

Once it worked, the brokerage's agents spent long sessions trying to break it, and I listened to a lot of calls. The mistakes that surfaced were behavioral, and most were fixed in the prompt. Don't ask "buy or lease?" when someone is clearly describing a land development. Don't ask for the city when they have just named a neighborhood. Read street numbers in pairs, "twenty twenty-five", never as one big number. Stop asking for a name after a polite "I'm good." Stop talking when someone says stop.

Your replies are read aloud over the phone by a text-to-speech voice.
Write everything the way it should be spoken. No abbreviations:
say "Street", not "St", and "Northwest", not "NW".
Split street numbers into pairs: 2025 is "twenty twenty-five",
1805 is "eighteen oh five". Never say "two thousand" for an address.
Say "square feet", not "sq ft".

Where the problem was the data, I fixed the data: standardizing addresses and cleaning inconsistent records. A model reading messy listings will never seem smart, however good it is.

Don't make people spell their email out loud

The client wanted listings emailed to callers. Dictating an email address to a voice agent is miserable, so we text the caller instead. That meant getting business texting approved with the carriers and building a consent model, the part that became the sms-consent library, part 2 of this series. The caller replies to the text with their address, and the flyer arrives by email.

If you build one

• Sent is not heard. Know what the caller actually heard, and make the agent's memory match.
• Be suspicious of every rule that discards input. Short answers carry the intent.
• Keep word-correction tables narrow and tied to the question being asked.
• Fix the data before you fix the prompt, and the prompt before you reach for code.
• Listen to real calls. Logs say what happened; only listening tells you how it felt.
• Never ask anyone to spell something out loud.

I built this with Claude as my pair, in the browser chat and later in Claude Code. I wrote the spec, listened to the calls, and decided what shipped. The turn-taking pieces are in the library, and the rest of this post is how to run it.

What's in the library

The part of the agent above that decides what the caller actually heard, packaged so you can read it in an evening. It has no opinion about your prompt, your tools or your CRM.

• Playback you can trust: only Twilio's mark echoes count as played, so a barge-in cuts the turn at what the caller really heard.
• Text-to-speech timing: map "milliseconds played" to "words heard" using per-character timings. A word the caller only partly heard counts as unheard.
• Memory rewrite: the interrupted assistant message becomes the heard part plus a marker, so the model's next answer is based on reality.
• Backchannels: "uh-huh" and "sí claro" do not interrupt; a real utterance does. The classifier is pluggable.

import { splitAtPlayedMs, rewriteOnBargeIn } from "@27colab/voice-engine";

const { spoken, unspoken } = splitAtPlayedMs(fullText, alignment, playedMs);
rewriteOnBargeIn(history, spoken, { inHistory: true });
// last assistant message: "Sure, I can help with that. We have [interrupted]"

Run it offline first

No keys, no phone number. The repository's loopback example simulates a whole call with fakes: the caller hears about two seconds, interrupts, and the model's next prompt shows the cut-off turn.

git clone https://github.com/27colab/voice-engine.git
cd voice-engine && npm ci
npm test          # the offline test suite
npm run example   # a simulated call, interruption included

Run it on a real phone number

You need four accounts. Each one bills on usage, so fund them with a small amount, set spending alerts, and read the dashboards after your first ten test calls.

1. Anthropic: create an API key in the console and add credit. The example uses Claude Haiku, which suits live calls because it is fast.
2. Deepgram: create an API key. The example uses its Flux model for turn-taking.
3. ElevenLabs: create an API key, then pick a voice (and a second one for Spanish if you want bilingual). Copy each voice's id.
4. Twilio: create an account, fund it, and buy a local phone number with voice capability. Twilio asks for identity and payment verification before a number can take real calls; do that in the console, never through a link in an email.

Put the keys in a .env file, using the repository's .env.example as the template:

DEEPGRAM_API_KEY=...
ELEVENLABS_API_KEY=...
ANTHROPIC_API_KEY=...
ELEVENLABS_VOICE_EN=<voice id>
ELEVENLABS_VOICE_ES=<voice id, optional>

Answer the call: the webhook is yours

When someone dials the number, Twilio makes an HTTP request to the number's Voice webhook and expects TwiML back. The library does not include that endpoint; the example only opens the WebSocket that the TwiML points at. For a first test, paste the XML below into a TwiML Bin in the Twilio console and set it as the number's "A call comes in" handler. As soon as you have more than one number or one business, return it from a route in your own server instead: custom <Parameter> tags on the stream arrive in the start event, which is how your WebSocket handler learns who was called before the first audio frame.

<Response>
  <Connect>
    <Stream url="wss://YOUR-HOST/media">
      <Parameter name="tenantId" value="acme" />
      <Parameter name="callerPhone" value="{{From}}" />
    </Stream>
  </Connect>
</Response>

Validate the X-Twilio-Signature header on that route before you trust anything in the request. Twilio signs the full public URL plus the sorted form fields with your auth token (HMAC-SHA1, base64); behind a load balancer, rebuild the URL from your canonical public origin or the signature never matches. The example skips this. A server on the public internet cannot.

Then start the example server:

npx tsx examples/live-twilio.ts      # listens on :8080/media

Testing from your laptop: use a tunnel

Twilio needs a public address, and your laptop doesn't have one. I lost a lot of time early on deploying to a hosting service on every change just to get a URL to call. A tunnel such as ngrok fixes that: run it against your local port, paste the wss address into the TwiML, and every change is one save and one phone call away.

ngrok http 8080      # copy the https host, use it as wss://<host>/media

Then call the number, talk over the agent, and read the call record it prints when you hang up.

Write the prompt: the engine has no opinion

The system prompt in the session config is a string, a list of blocks, or a function of the current language and time that returns either. The example's prompt is one sentence. A receptionist that a business will put its name on needs much more, and all of it is yours to write:

• Rules of the medium: the reply is spoken by a text-to-speech voice, so no markdown, no lists, short sentences, numbers and times written the way they should be said, street numbers in pairs.
• Safety and scope: what to do with an emergency, no medical or legal advice, never send anything to a third party the caller names, when to end an abusive call.
• Business facts and policies: the name as it is pronounced, hours in the local time zone, services, prices the owner approved, and what the agent must never invent.
• What each tool does and when to use it, in the model's terms, plus the disclosure that the call is recorded and the caller is talking to an AI.
• Volatile facts: the current local time and whether the business is open right now. Without them the model will happily invite people to walk in at a quarter to ten at night.

You are answering a live phone call as the business's receptionist.
Your words are spoken aloud by a text-to-speech voice: no markdown, no
bullet points, no emojis. One to three sentences, then stop.
Read prices and times naturally ("nine thirty in the morning").
If a transcription seems garbled, ask the caller to repeat; never
mention "transcription". Never invent facts about the business; offer
to take a message instead.

[safety rules]
[business facts, hours, services]
[what you can do on this call: tools]
--- cache boundary ---
Current date and time at the business: Tuesday, 9:42 PM. CLOSED now;
next opening is tomorrow at 9:00 AM.

Write the stable parts first and the volatile parts last. That order is the whole trick in the next section.

Prompt caching: on by default, with one rule

Every turn of a call sends the whole system prompt and the whole conversation again. For a receptionist prompt of a few thousand tokens, over a dozen turns, that is most of the input cost of the call and a measurable slice of the silence before the first word. Anthropic's prompt caching fixes it: mark a boundary, and on the next request everything before it (the tool definitions, then the system prompt, in that order) is served from cache at a fraction of the price and without being re-read.

Since version 0.2 the adapter that ships with the library does this for you: it caches the system prompt, with the tool definitions in front of it, and the conversation so far. The one rule is order. Pass the prompt as blocks, mark the last stable one with cacheBoundary, and put the volatile block after it, so a changing clock never invalidates the cached prefix. I first wrote this as a hundred-line chat engine beside the library when I put it under a multi-tenant platform; it belonged inside, so it moved.

import { createCallSession } from "@27colab/voice-engine";
import { createAnthropicChat } from "@27colab/voice-engine/providers";

// Caches the system prompt (with the tools in front of it) and the conversation so far.
const chat = createAnthropicChat(process.env.ANTHROPIC_API_KEY!);

const session = createCallSession(
  { transport, sttFactory, ttsFactory, chat },
  { callId, startedAt: new Date() },
  { onEnd: (record) => meter(record.stats.usage) }, // tokens incl. cache reads/writes, TTS characters
  {
    // Stable first, volatile last. Everything up to the boundary must be byte-identical
    // from turn to turn (and from call to call for the same business).
    system: (lang, now) => [
      { text: craftRules },
      { text: safetyRules },
      { text: businessFacts(lang) },
      { text: toolInstructions, cacheBoundary: true },
      { text: liveFacts(now) }, // after the boundary: time, open or closed
    ],
    // ...languages, greeting, llm, tools, voice, stt
  },
);

• Keep the prefix byte-identical: same tool list in the same order, same block order, no timestamps or call ids before the marker. Sort anything you assemble from a database.
• There is a minimum cacheable prefix, and it depends on the model. Below it the marker is ignored silently, with no error. On Claude Haiku 4.5 it is 4,096 tokens: a receptionist prompt of about 3,800 never cached until the adapter also cached the conversation, which crosses the floor from the second turn. If cache reads stay at zero after that, something in your prefix changes per request.
• The default cache entry lives five minutes from its last use, which a live call refreshes on every turn. A one-hour option exists for numbers that ring less often; it costs more to write.
• The call record carries the totals: record.stats.usage has input, output, cache-read and cache-write tokens and text-to-speech characters, ready to price per call.

Tools, fillers and the sound of waiting

The engine runs the tool loop: the model asks for a tool, the engine calls your executeTool, hands the result back, and continues until the model answers in words. What each tool does, searching listings, checking a calendar, logging a lead, is yours, and so is making it fast. A caller hears every millisecond of a slow API.

A two-second lookup in silence sounds like a dropped call. Give the session a filler per tool, "one moment, let me check that", in each language you support, and the engine speaks it the instant the model starts the call. It plays through the same playback path as everything else, so the caller can interrupt it, and it never lands in the model's history. It is skipped when the model already said its own preamble. The phrase is yours to write, because it belongs to the business.

Recording, disclosure and the parts that are not code

• <Connect><Stream> does not record. If you need recordings, start one through Twilio's REST API when the call begins, and say in the greeting that the call is recorded and the caller is speaking with an AI assistant. Several US states require the disclosure, and callers deserve it anyway.
• The caller's number arrives as a webhook field, not over the stream. Pass it through a <Parameter> if the agent needs it, for example to text them afterwards.
• Texting the caller afterwards is its own registration (A2P 10DLC in the United States) with a lead time of weeks. That is part 2 of this series.
• Nothing here is multi-tenant or persistent. The engine totals usage per call but does not price or store it. A platform with many businesses on it needs the prompt assembled per business before the stream starts, call records written somewhere, and usage attributed per business. All of that sits above the engine.

What this does not do

• The example server in the repository (examples/live-twilio.ts) is a demo, not a deployable server: no authentication, no Twilio signature validation, no concurrency limits.
• Played time is a lower bound, so the rewrite errs toward "heard less".
• Single process, in-memory, pre-1.0. Nothing here has been benchmarked for latency or quality.
• No prompt assembly, no persistence, no multi-tenancy, no billing. Each of those is a short adapter in the platform I built on top of it, and the sections above say what each needs.
• It is a compact reference for the parts that are easy to get subtly wrong, not a platform. Vapi, Retell, LiveKit Agents and Pipecat already ship barge-in.

Get it

MIT-licensed, early (v0.2), written with Claude Code and directed and reviewed by me. Install from npm where published, otherwise clone the repository.

git clone https://github.com/27colab/voice-engine.git
cd voice-engine && npm ci && npm test