Live English Tutor: Building a Real-Time Voice AI English Tutor with LiveKit + Gemini Native Audio

Picture this: instead of typing into a chatbot to practice English, you just... talk. Out loud, through your microphone, the same way you'd talk to a real tutor sitting across the table. That's the whole idea behind Live English Tutor — a real-time voice AI system where students have actual spoken conversations with an AI teacher named Emma. She listens, she talks back, she corrects your mistakes on the fly, and when the session's over, she hands you a written learning report in Chinese.

Now, most language-learning apps out there are text-first. You read, you tap, you fill in blanks. This project deliberately went the other direction — natural spoken conversation as the core experience. And here's the thing: the moment you commit to real-time voice, you've signed up for the single hardest engineering problem in the whole system, which is latency. How do you make the AI respond fast enough that it feels like a conversation and not a walkie-talkie exchange with awkward pauses? That question drives basically every design decision here.

Let me walk you through how it's built, because the architecture is genuinely clean. The system is split into three decoupled layers, each minding its own business.

First, the frontend — that's the browser. Built with React and Vite. It handles Google Sign-In through Firebase, talks to the backend over a normal REST API, and crucially, it establishes the live voice and video connection using the LiveKit JavaScript SDK. That's your media pipe.

Second, the backend — a FastAPI service. This is the traffic controller. It verifies your login token from Firebase, manages your courses and messages, issues the tokens that let you into a LiveKit room, and saves everything into a PostgreSQL database.

And third, the star of the show: the LiveKit Agent Worker, which *is* Emma. She runs as her own standalone worker, powered by Google's Gemini 2.5 Flash Native Audio model. When you're in a session, the audio and video stream flows from LiveKit straight into Emma, and Emma streams her voice right back.

Around all this sit a few external services — LiveKit itself running self-hosted in Docker, Firebase for auth, the Google Gemini API for the actual conversation and voice, and Ollama running on a separate server to generate those post-class reports.

So why Gemini Native Audio? This is the key decision, so let me slow down here. The traditional way to build a talking AI is to chain together a whole assembly line: first voice activity detection to figure out when you're speaking, then speech-to-text to transcribe what you said, then a language model to think up a response, then text-to-speech to say it out loud. Four separate stages. And every single one adds a little delay. Stack them all up, and the conversation starts to feel sluggish — that fluency just falls apart.

Gemini 2.5 Flash Native Audio collapses all four of those stages into one integrated model. Voice goes in, voice comes out, and all the detection, understanding, and speaking happen natively inside. That's a dramatic cut in both latency and integration complexity. And because it's configured with video enabled, Emma can also see your camera feed or your shared screen — not just hear you.

Now, Emma isn't just one static personality reading from a single script. She runs on a four-stage state machine, and she moves through the stages in order, with a different system prompt for each. She starts in WARMUP — easing you in, warming up the conversation. Then she moves to PRACTICE, the main conversation drills. From PRACTICE she can dip into CORRECTION when you make a mistake — real-time correction — and then loop right back to PRACTICE to keep going. And finally, when the lesson's winding down, she transitions to SUMMARY. Throughout all of this, every message and every correction gets saved. Emma's worker calls back to the backend's internal API to persist them — and those internal endpoints are locked down with a special secret header, meant only for the Agent, completely walled off from the user-facing API.

Let me trace one full class start to finish, because it ties everything together. You sign in with Google. The frontend asks the backend to create a session and hand over a LiveKit token. The backend spins up a room on LiveKit and dispatches the Agent — that boots Emma. Your browser joins the room over WebRTC. And now you're in the loop: you speak, Emma responds and corrects in real time, and behind the scenes she's saving each exchange to the backend. When you're done, you end the session, the frontend notifies the backend, and the backend triggers Ollama to generate your post-class report. That report comes back in Chinese.

One detail worth knowing: report generation is deliberately kept independent from the main conversation, and it's actually turned off by default. You have to flip an environment flag to true and confirm your Ollama server is reachable before it'll run. The frontend can poll for status — is the report disabled, pending, or ready — and display accordingly.

Now I want to talk about the self-hosting, because there's a war story here that anyone who's touched WebRTC will recognize. LiveKit runs fully self-hosted in Docker — no LiveKit Cloud account needed, the whole stack comes up locally with one compose file. Great. But then the classic bug hit: everything worked perfectly on the local machine, and the moment you tried to connect from another device on the same network, the connection just... failed. "Could not establish PC connection."

Here's what was going on. WebRTC uses these things called ICE candidates to figure out how two devices should reach each other — essentially a list of "here's where you can find me." And LiveKit was broadcasting the *wrong* IP address in that list. The fix was to explicitly set the node's IP to the host's actual LAN address and restart the server. Once LiveKit advertised the correct address, cross-device connections worked. And the broader lesson — file this away — when something works locally but drops the instant you switch devices, the culprit is almost always the ICE and networking layer, not your application code. Don't waste hours debugging your business logic on that one.

A quick word on security, because the system has two crystal-clear trust boundaries. On the outside, all the user-facing APIs require a valid Firebase ID token — you sign in with Google first, get your token, and the backend verifies it with the Firebase Admin SDK. On the inside, the Agent-to-backend callbacks are guarded by that shared secret header I mentioned, totally isolated from the external API. And two easy-to-forget deployment gotchas: after you push the frontend to Cloudflare Pages, you have to add that pages.dev domain to Firebase's authorized domains, or Google Sign-In throws an unauthorized-domain error. And you have to add the frontend's domain to the backend's allowed origins list, or CORS will block you.

So let me leave you with the three things that really matter here.

One: the architecture wins by decoupling. Media, API, and AI Agent are three independent layers, each swappable and operable on its own. That separation is what makes the whole thing tractable.

Two: the native audio model is the make-or-break decision. Replacing that traditional four-stage speech pipeline with a single integrated Gemini model is what buys you low enough latency for the conversation to actually feel natural. If you take one technical idea away, take that one.

And three: the boring operational stuff is what keeps it alive. Self-hosted LiveKit, Firebase auth, the internal secret, and that ICE networking fix — none of it is glamorous, but together it's the foundation that lets this run in both your local setup and production without falling over. Real-time voice AI is exciting, but it's the plumbing that determines whether it ships.