To Help My Socially Anxious Fans, I Built an AI Video Calling Product

Picture this. You're a tech YouTuber with a big following, and one day a comment lands in your inbox that stops you cold. Someone writes: "I'd love to ask you questions, but I have social anxiety — even typing this feels stressful, let alone getting on a real call." Now, most people would reply with some encouraging words and move on. This person did something different. He decided to build a product.

So today I want to walk you through the engineering behind an AI video-calling app — a place where you can practice real conversation, except the person on the other end of the call isn't a person at all. It's AI. And I want to focus on the decisions an indie developer has to make when they're building something like this with limited time, limited money, and a very hard technical problem at the core.

Let's start with the why. Social anxiety disorder affects roughly seven percent of adults worldwide. At its heart is an outsized fear of being judged, which makes face-to-face conversation — and especially video calls — genuinely stressful. The classic treatment is cognitive behavioral therapy, and specifically something called exposure therapy: you gradually, gently expose someone to the situations that scare them. The theory is solid. But in the real world it hits walls. Therapists are expensive and booked out for months. You can't find a willing human to practice small talk with every single day. And when you do practice with a real person and it goes badly, that failure can actually deepen the anxiety.

So here's the hypothesis behind the product. What if you gave people a low-risk practice space? The conversation partner is AI. If you freeze up, if you fumble, if you say something awkward — you just restart. Nobody's judging you. Nobody even knows.

For that to work, the experience has to clear four bars. First, it has to *look* like a real person is talking to you — not a text box, not a faceless voice. Second, it has to be fast. If the AI takes more than a second to respond, the rhythm of conversation collapses. Third, it has to actually hold a conversation — remember what you said earlier, track context. And fourth, it has to feel safe, which means being honest that it's AI. You never want the user to feel tricked.

Now let's get into the stack, because this is where the trade-offs live.

Start with the video itself. The natural choice is WebRTC, because it's built right into the browser — no app to download — and peer-to-peer connections are fast, under fifty milliseconds in the same region on a good day. But here's the catch for a solo developer: running your own WebRTC signaling and TURN servers is a real chunk of work. So the practical move is a managed service. You've got Daily.co, which is roughly four-tenths of a cent per participant per minute and gets you shipping fastest. There's Livekit, which is open source and self-hostable with a generous free tier. And there's Agora, which is enterprise-grade and rock-solid but comes with more complicated pricing. For a first version, you lean on the managed option and save yourself the infrastructure headache.

Next, the hard one: giving the AI a face. This breaks into three tiers. The simplest is a static image plus audio — a still picture that doesn't even move its mouth, just talks. Dead easy to build, but the experience is flat. The middle tier is a 2D avatar with lip-sync, using something like Ready Player Me, where you map the audio waveform to mouth movements. Cheap, works, but it looks like a virtual streamer, not a human. And then the top tier: real, AI-generated video. Services like Tavus and HeyGen let you upload a video of an actual person, and then generate a live speaking video stream in real time that matches whatever the AI is saying. The latency runs somewhere between three hundred and eight hundred milliseconds, and the result is the closest thing to talking to a real human. Tavus even has a Conversational Video Interface built specifically for this. The smart play for an indie dev? Validate the idea with the cheap tiers first, and only upgrade to real-time video once you've got users who care.

Then there's the pipeline that actually makes a conversation happen. You speak, and the system has to hear you. That's speech recognition — ASR. Whisper from OpenAI is accurate but too slow for live use, over half a second of latency. Deepgram is tuned for real-time streaming and comes in under two hundred milliseconds. So for this kind of product, Deepgram or AssemblyAI is the go-to.

Your words become text, and that text goes to a language model to generate a reply. Here, latency is everything, so you want models that start talking fast — think GPT-4.1, Claude Haiku 3.5, or Gemini Flash. And this is where a huge amount of the product magic lives: the system prompt. That's what tells the AI to *be* a job interviewer, or a stranger you're asking for directions, or someone you're calling on the phone — and to pace itself naturally.

Finally, the model's reply, still just text, has to become a voice. That's text-to-speech. ElevenLabs gives you the best audio quality with real-time streaming. OpenAI's TTS is simple and solid. Play.ai is designed for conversation and can adjust emotional tone.

Now here's the part that keeps engineers up at night. Add up that whole chain — you finish speaking, then two hundred milliseconds for recognition, three hundred for the model to start replying, two hundred for the first chunk of audio. That's around seven hundred milliseconds, best case. And seven hundred is already living on the edge. The moment any single link hiccups — a network wobble, the model under load — you blow past that one-second threshold where it stops feeling natural. This latency budget is genuinely the hardest engineering problem in the whole product.

A couple of supporting pieces make it work. There's voice activity detection — the system needs to know the *exact* moment you've stopped talking. Trigger too early and the AI cuts you off mid-sentence. Trigger too late and you add lag. Silero VAD is a popular open-source tool for this. And there's memory: language model APIs are stateless, they forget everything between calls, so you have to manage the conversation history yourself — keep the last few turns in full, summarize the older ones to save on tokens, and stash the important facts in a vector database.

Quick word on the market, because it tells you something. AI companion apps crossed 220 million downloads in 2025, and the number of those apps grew seven hundred percent in just two years. The company behind the virtual pet Pengu raised a fifteen-million-dollar Series A specifically for social AI companions. So the demand is clearly there — but so is the competition. And the lesson for a small builder is: don't try to build a general-purpose AI chat buddy. Go deep on one specific scenario. Just interview practice. Just phone-call anxiety. That focus is your edge.

So let me leave you with the three things worth remembering.

One: latency is the whole game. Conversation feel is subjective, sure, but push response time past eight hundred milliseconds and almost everyone can feel it. Get that wrong and nothing else matters.

Two: looking real beats being feature-rich. Users don't care how long your feature list is. They care about one thing — does this feel like a real conversation?

And three: build the cheap version first. An MVP stitched together from ElevenLabs, Deepgram, and Claude costs very little and answers the only question that matters early on — will people actually use this? Because in the end, the deepest differentiator here isn't the infrastructure. It's the prompt engineering that makes the AI feel human. That's the part no managed service can hand you.

And that's the real story. A single fan message, a hard latency problem, and a builder who decided the best reply wasn't advice — it was a tool.