agentic.dev

AI Voice Agents: Phone Bots with Twilio and ElevenLabs

Published 2026-02-21

AI Voice Agents: Phone Bots with Twilio and ElevenLabs

For years, the promise of truly intelligent voice assistants on the phone has felt…distant. IVR systems (Interactive Voice Response) were, and largely are, brittle decision trees masquerading as conversation. Then came the early days of speech-to-text and text-to-speech, hampered by robotic voices and poor accuracy. But in 2026, we’re finally at an inflection point. Large Language Models (LLMs) coupled with realistic voice cloning and synthesis technologies – specifically, Twilio for telephony and ElevenLabs for voice – make building genuinely useful and engaging AI voice agents a practical reality. I’ve spent the last quarter building a sophisticated appointment scheduling bot using this stack, and I want to share the practical details, the surprising costs, and the architectural decisions that made it work. Forget the “press 1 for…” experience; we’re building bots that understand and respond like a human.

Why Twilio and ElevenLabs? The Modern Stack

The core challenge in building an AI voice agent isn’t just the AI part. It’s reliably connecting to the phone network, handling call flow, and delivering audio that doesn't immediately scream “robot”. This is where Twilio shines. They provide the Programmable Voice API, a robust and well-documented system for managing phone calls programmatically. It handles the complex details of SIP signaling, DTMF tones, and global phone number provisioning. Alternatives like Vonage exist, but Twilio’s developer experience and widespread adoption, particularly within the AI/agentic development community, make it the default choice for many.

However, Twilio’s native text-to-speech (TTS) voices, while improved, still lack the nuance and realism required for a truly convincing experience. This is where ElevenLabs steps in. ElevenLabs isn’t simply TTS; it’s voice cloning and expressive speech synthesis. You can upload a sample of a voice (with proper permissions, of course!), and ElevenLabs will create a digital replica capable of speaking almost anything. More importantly, their API allows for control over emotional tone, speaking style, and even subtle vocal inflections.

The combination is potent. Twilio handles the plumbing, and ElevenLabs provides the personality. You could theoretically use other voice synthesis APIs – Microsoft Azure AI Speech, Google Cloud Text-to-Speech – but in my testing, ElevenLabs consistently delivered the most human-sounding results, justifying the cost (more on that later). The key to a good agent is believability, and ElevenLabs makes believability far more attainable.

Architecting the Conversation Flow: From Speech to Action

Here's how I structured the call flow for the appointment scheduling bot. It’s a simplified view, but illustrates the core components.

  • Incoming Call (Twilio): A user dials the Twilio phone number.
  • Speech-to-Text (Twilio or Whisper API): Twilio's built-in STT (Speech-to-Text) is a decent starting point, but for higher accuracy, especially with varied accents and background noise, I opted to stream the audio to OpenAI's Whisper API. It's more expensive, but the improvement in transcription quality is significant.
  • LLM Processing (GPT-4o via API): The transcribed text is sent to GPT-4o (the current state-of-the-art in 2026). The prompt is crucial. I used a carefully crafted prompt that included:
    • * System Message: Defining the bot's role (“You are a friendly and efficient appointment scheduling assistant for Dr. Anya Sharma.”). * Conversation History: Maintaining context across multiple turns. * Intent Recognition: Asking the LLM to identify the user's intent (e.g., “schedule appointment”, “cancel appointment”, “ask about availability”). * Entity Extraction: Extracting relevant information (e.g., date, time, appointment type).
  • Action Execution: Based on the LLM’s analysis, the bot performs an action. This could involve:
    • * Querying a database for available appointment slots. * Updating the database with a new appointment. * Sending a confirmation SMS message via Twilio.
  • Text-to-Speech (ElevenLabs): The bot generates a response using the LLM. This text is sent to ElevenLabs for voice synthesis. I used a cloned voice of a professional receptionist for a consistent and welcoming tone.
  • Audio Playback (Twilio): The synthesized audio is streamed back to the user via Twilio.

    • Here’s a snippet of Python code (using the Flask framework) demonstrating the core interaction with Twilio and ElevenLabs:

    from flask import Flask, request
from twilio.twiml.voice_response import VoiceResponse, Say
import requests
import os

    app = Flask(__name__)
    

    ELEVENLABS_API_KEY = os.environ.get("ELEVENLABS_API_KEY") VOICE_ID = "your_cloned_voice_id" # Replace with your ElevenLabs voice ID
    

    @app.route("/voice", methods=['POST']) def voice():     """Handles incoming voice calls from Twilio."""     response = VoiceResponse()
    

    # Get user input (digit or speech)     digit_pressed = request.values.get('Digits', None)     if digit_pressed:         user_input = digit_pressed     else:         # In a real-world scenario, you'd use <Gather> to collect speech         # and then transcribe it using Whisper.  For simplicity, we'll         # assume we receive pre-transcribed text from Twilio (or Whisper).         user_input = request.values.get('Speech', "Please say something.")
    

    # Send user input to LLM (replace with your actual LLM call)     llm_response = get_llm_response(user_input)
    

    # Synthesize speech with ElevenLabs     try:         payload = {             "text": llm_response,             "voice": VOICE_ID,             "model": "eleven_turbo_v1" #Current best model in 2026         }         headers = {             "Accept": "audio/mpeg",             "Content-Type": "application/json",             "xi-api-key": ELEVENLABS_API_KEY         }         response_url = "https://api.elevenlabs.io/v1/text-to-speech/turbo"         response = requests.post(response_url, json=payload, headers=headers, stream=True)         response.raise_for_status() # Raise HTTPError for bad responses (4xx or 5xx)                  #Play the audio back to the user         response_twiml = VoiceResponse()         response_twiml.play(url=response.url)         return str(response_twiml)
    

    except requests.exceptions.RequestException as e:         print(f"Error calling ElevenLabs API: {e}")         response_twiml = VoiceResponse()         response_twiml.say("Sorry, there was an error processing your request.")         return str(response_twiml)
    

    def get_llm_response(user_input):     """Placeholder for LLM interaction.  Replace with your API call."""     # In a real application, you would send the user_input to your     # LLM (e.g., GPT-4o) and receive a response.     return f"You said: {user_input}.  I'm processing your request..."
    

    if __name__ == "__main__":     app.run(debug=True)

    Important Considerations:

    The Cost of Believability: A Realistic Breakdown

    Building these agents isn’t cheap. Here’s a breakdown of the costs I encountered:

  • Twilio: Pay-as-you-go pricing. Costs include phone number rental (around \$1/month), incoming call charges (typically \$0.01/minute), and outgoing call charges (variable based on destination). Using Twilio’s STT adds about \$0.005/minute. For a moderate volume of calls (1000 minutes/month), Twilio could cost around \$150 - \$250.
  • ElevenLabs: Pricing is based on character count. As of late 2026, their premium voice cloning and expressive synthesis is approximately \$0.015/character. A typical conversation turn might generate 200-300 characters. 1000 minutes of conversation could easily generate 1.5 million characters, costing around \$2250. This is the biggest cost driver.
  • OpenAI (Whisper & GPT-4o): Whisper costs around \$0.006/minute for transcription. GPT-4o is priced by token usage; a typical conversation turn might consume 500-1000 tokens, translating to around \$0.02 - \$0.05 per turn. 1000 minutes of conversation with LLM and Whisper could cost \$500 - \$1000.
  • Infrastructure (Redis, EC2): Running the application on AWS EC2 and using Redis for context storage adds another \$50 - \$100/month.
  • Development & Maintenance: Don't forget the cost of your time! Developing and maintaining a complex AI agent requires significant engineering effort.

    • Total estimated cost (1000 minutes/month): \$3125 - \$3850.

    This highlights a crucial tradeoff: believability comes at a price. While costs are decreasing, building a truly sophisticated AI voice agent is still a substantial investment.

    Future Proofing: Considerations for 2027 and Beyond

    The AI landscape is evolving rapidly. Here are a few things I’m keeping an eye on:

  • Multimodal LLMs: LLMs that can process both text and audio will simplify the architecture and potentially improve accuracy.
  • RAG (Retrieval-Augmented Generation) for Voice: Using RAG to ground the LLM in a specific knowledge base will be crucial for building agents that can handle complex queries.
  • Voice Activity Detection (VAD) improvements: More accurate VAD will reduce unnecessary processing and costs.
  • Standardised Agent Frameworks: The emergence of frameworks like AutoGen and CrewAI are starting to abstract away the complexities of managing multiple AI models. Expect these to become more mature and voice-focused.
  • Cost Optimisation: Exploring techniques like voice compression and model quantization to reduce the cost of ElevenLabs and OpenAI API calls will be vital.