Voice Agents — Part 3: Building a Realtime AI Mock Interviewer

In the previous article, we looked at how realtime voice architectures are evolving: from classic STT / LLM / TTS pipelines to streaming pipelines, conversation-control layers, audio-capable models, and native speech-to-speech systems.

In this article, I want to move from architecture patterns into a concrete product case: an AI mock interview system. You can take a look at final product here Koffi Interview

A mock interview is a strong use case for voice agents because the product value depends heavily on interaction quality. The candidate needs to speak, listen, think under pressure, answer follow-up questions, and review their performance afterward.

Product Context

A mock interview has a natural conversation structure.

The session may start with a short introduction, move into the candidate’s background, continue with role-specific technical questions, branch into follow-up questions, and end with evaluation and feedback.

The important part is that the interview should not feel fully scripted.

If the candidate says:

I built a Kafka-based event processing pipeline.

The system should not simply move to the next question.

It should be able to go deeper:

How did you design the partitioning strategy, and what tradeoffs did you consider?

Or:

You mentioned consumer lag during traffic spikes. How did you detect it, and what recovery strategy did you use?

That follow-up behavior is what makes the session feel closer to a real interview.

For this product, voice matters because interview performance is not only about answer correctness. It also includes clarity, pacing, confidence, structure, and communication.

Those are difficult to practice through text alone.

Architecture Overview

The architecture has four major layers:

• Client application

• Realtime media layer

• Voice agent backend

• Interview intelligence layer

At a high level, the flow looks like this:

Cloudflare TURN is part of the connectivity layer. TURN means Traversal Using Relays around NAT. It provides a relay path when direct WebRTC connectivity is blocked or unreliable because of firewall or NAT restrictions.

The main architectural idea is separation of responsibility.

The media path needs to move voice reliably and with low latency. The backend needs to understand the interview session, process speech, generate interviewer responses, and store useful review data.

Those two problems are connected, but they are not the same problem.

Keeping them separated makes the system easier to reason about and easier to evolve.

Client Layer

The client is the user-facing interview room.

In practice, this means the browser or mobile app is responsible for capturing the candidate’s microphone, playing the interviewer’s voice, maintaining the WebRTC connection, and showing realtime UI feedback.

WebRTC is used for the live audio session. ICE, or Interactive Connectivity Establishment, is used by WebRTC to find the best available network path for the media connection.

The client also needs to show the interview state clearly. The candidate should know whether the interviewer is listening, thinking, speaking, or reconnecting. The UI may also show the current question, live transcript, and final feedback after the session.

Media Layer: WebRTC and Janus

WebRTC provides the realtime audio connection from the browser or mobile client.

Janus acts as the media gateway and audio room.

The practical reason for using Janus is that the product needs a controlled realtime media layer between the candidate and the AI backend.

A useful mental model is:

The candidate joins a realtime audio room, and the AI interviewer participates through the backend media path.

This model gives the system room to grow.

Today, the room may contain one candidate and one AI interviewer. Later, the same media model can support observer mode, recording, replay, human review, different interviewer voices, or more advanced session monitoring.

Janus also gives a cleaner boundary between the client and backend. The browser or mobile app can focus on the WebRTC session and user experience, while the backend can focus on consuming audio, generating responses, and coordinating interview state.

This is the main architectural benefit.

Connectivity Layer: TURN

In development or ideal network conditions, WebRTC may connect directly without much trouble.

Real user networks are different.

A candidate may join from a home router, corporate network, VPN, mobile network, university network, or a restrictive firewall environment. Some networks block or degrade UDP, which stands for User Datagram Protocol. WebRTC prefers UDP for realtime media because it avoids some of the delays caused by reliable ordered delivery.

When direct media connectivity fails, TURN provides a relay path.

This is where Cloudflare TURN fits.

From an infrastructure point of view, TURN is part of NAT traversal.

From a product point of view, TURN is part of reliability.

The candidate does not know whether the problem is ICE negotiation, firewall policy, NAT behavior, UDP blocking, or poor routing. They only experience that the interview does not connect, audio is one-way, or the session feels unstable.

For a realtime mock interview, that is enough to break trust.

Voice Agent Backend

The voice agent backend is where the realtime media stream becomes an interview experience.

It coordinates incoming audio, streaming transcription, interview state, LLM response generation, text-to-speech generation, outgoing audio, interruption handling, session persistence, and evaluation data.

A practical backend usually needs several internal responsibilities:

• Audio stream handling

• Speech-to-text adapter

• Transcript manager

• Interview session orchestrator

• LLM gateway

• Retrieval and context service

• Memory/session store

• Text-to-speech adapter

• Evaluation pipeline

The important part is coordination.

The backend is not only calling speech-to-text, then an LLM, then text-to-speech. It is managing a live interview session.

For example, when the candidate finishes an answer, the backend needs to decide whether the answer is complete, whether a follow-up is useful, whether the system should move to the next interview stage, and what should be stored for review later.

This is where a mock interview product becomes more specific than a generic voice assistant.

Interview Intelligence

The interview intelligence layer is the part that turns a voice conversation into a structured interview session.

For Zamo’s mock interview product, the LLM does not receive only the latest sentence from the candidate. It receives controlled context built from the interview plan, candidate profile, target role, previous answers, retrieved documents, and the current transcript segment.

That context is what allows the interviewer to ask relevant questions instead of generic ones.

Interview Plan

Each session follows an interview plan.

The plan defines the broad stages of the conversation: introduction, experience review, role-specific questions, technical deep dive, behavioral discussion, closing, and review.

The plan is not a rigid script.

It gives the interviewer direction while still allowing the session to branch based on the candidate’s answers.

For example, if the candidate gives a shallow answer, the system may ask for clarification. If the candidate mentions a specific technology, the system may go deeper. If the candidate already covered a topic well, the system can move forward instead of repeating the same area.

Without this structure, the conversation can become random.

With too much structure, the session becomes scripted.

The product needs a balance between interview flow and adaptive conversation.

Candidate and Role Context

The interview is grounded in the candidate profile and target role.

When the user provides a CV or interview profile, the system extracts useful context such as experience, projects, skills, seniority signals, and technologies mentioned.

When the interview is created from a job description, the system extracts expected skills, topic areas, and evaluation criteria.

This changes the interview behavior.

A senior Java backend interview should not feel the same as a junior frontend interview. A candidate who mentions Kafka, Redis, Kubernetes, or PostgreSQL should receive questions that match those areas. A role that emphasizes distributed systems should lead to deeper questions about scaling, failure handling, consistency, and observability.

The goal is not to make the interview artificially complex. The goal is to make it relevant.

Retrieval and Session Memory

Retrieval-Augmented Generation, or RAG, is useful when the interview needs to use longer context such as a CV, job description, question bank, or evaluation rubric.

In the mock interview flow, retrieval helps bring only the relevant context into the current LLM call instead of sending every document into every request.

For example, when the candidate is answering a question about Kafka, the system can retrieve related CV snippets, job requirements, or rubric items that are relevant to distributed systems and event streaming.

Session memory tracks what has already happened in the interview.

It includes the questions already asked, answers already given, skills mentioned, areas that looked weak, and topics that may deserve follow-up.

This matters because the interviewer should not react only to the latest transcript segment.

It should understand the interview so far.

Follow-Up Decision

Follow-up questions are one of the main signals of interview quality.

A weak follow-up sounds generic:

Can you tell me more about that?

A better follow-up is specific:

How did you design the Kafka partitioning strategy, and what tradeoffs did you consider?

A stronger follow-up reacts to the session context:

You mentioned consumer lag during traffic spikes. How did you detect it, and what recovery strategy did you use?

That difference comes from combining the transcript, interview plan, candidate context, retrieved information, and session memory.

Timing also matters.

If the system asks too early, it may interrupt the candidate’s thought process. If it waits too long, the session feels slow. If it relies on unstable partial transcripts, it may ask the wrong question.

This is why follow-up generation is not only prompt engineering.

It is a session orchestration problem.

Realtime Data Flow

In the mock interview system, not all realtime data should travel through the same path.

We separate the runtime flow into three paths:

• Audio path

• Transcript and UI data path

• Control and signaling path

This separation makes the architecture clearer.

Voice needs low-latency media transport. Transcript and UI updates need lightweight realtime messaging. Session setup and Janus coordination need signaling/control APIs.

Audio Path

The audio path carries the actual voice conversation.

The candidate’s microphone is sent through a WebRTC media track to Janus. Janus manages the audio room and routes the media stream toward the voice agent backend. After the backend produces the AI interviewer’s spoken response, the generated audio is sent back through Janus and played by the browser or mobile client.

This path is sensitive to latency, jitter, packet loss, reconnection behavior, and audio buffering.

Jitter means variation in packet arrival time. In voice communication, too much jitter can make audio feel unstable or delayed.

If the audio path is unstable, the interview feels unstable.

Transcript and UI Data Path

The transcript path is different from the audio path.

For Zamo’s mock interview product, live transcript updates can be delivered through a WebRTC data channel.

A WebRTC data channel allows non-audio messages to travel alongside the WebRTC session.

This is useful because the candidate’s voice can travel through the WebRTC audio track, while live transcript updates can travel as small realtime data messages.

For example:

• Partial transcript

• Final transcript

• Candidate speaking state

• AI interviewer status

• Current question update

This makes the interview room feel more transparent.

The candidate can hear the interviewer, see what was transcribed, follow the current question, and later review the conversation more easily.

Control and Signaling Path

Control events are different again.

Session setup, room joining, SDP exchange, ICE candidates, Janus plugin control, reconnect logic, start interview, and end interview belong to the control/signaling path. SDP means Session Description Protocol. It describes media capabilities such as codecs, network information, and how peers should connect.

Depending on implementation, this path can use Janus signaling over WebSocket or HTTP, together with the application backend’s own control API.

This gives a cleaner separation:

• WebRTC media tracks carry voice.

• WebRTC data channel carries live transcript and lightweight room state.

• Signaling/control channels manage session setup, Janus coordination, and interview lifecycle.

That separation is easier to debug than treating all realtime communication as one generic stream.

Runtime Challenges

The hardest problems in the mock interview system appear at runtime.

The architecture may look clean on paper, but the live session exposes timing, network, audio, and orchestration issues.

Latency Budget

Latency is the sum of many small delays.

It includes browser audio capture, WebRTC transport, Janus routing, backend processing, speech-to-text response time, LLM response time, retrieval latency, text-to-speech generation, buffering, and playback.

For a mock interview product, the useful metric is closer to voice-to-voice delay:

How long does it take from the candidate finishing a thought to hearing the interviewer respond?

That is what the user experiences.

This also means optimization must happen across layers.

Improving the LLM response time helps, but it may not be enough if text-to-speech has high time-to-first-audio.

Improving speech-to-text helps, but it may not be enough if turn detection waits too long.

Improving the WebRTC media path helps, but it may not be enough if backend orchestration is slow.

A voice interview system needs an end-to-end latency budget.

Interruption Handling

Interruption handling is one of the hardest parts to make natural.

In a mock interview, the candidate may interrupt the AI interviewer because they misunderstood the question, want to clarify something, realized they answered incorrectly, or started speaking before the interviewer finished.

A simple system may continue playing the AI voice until the end.

That feels unnatural.

A better system should detect candidate speech, stop or fade out the current audio, cancel the current response if needed, and return to listening.

This touches several layers.

Voice Activity Detection detects speech activity. Playback logic stops the current audio. Text-to-speech generation may need to be cancelled. LLM generation may need to be cancelled. Session state must be updated so the next response understands what happened.

The hard part is not only detecting sound.

The hard part is deciding what the sound means for the interview.

Was it an intentional interruption? Was it background noise? Was it a short acknowledgment? Was the candidate continuing the answer?

The product behavior should be different in each case.

Lessons Learned

A few concrete lessons came from applying realtime voice architecture to mock interview.

Media Reliability Is Product Quality

WebRTC, Janus, TURN, NAT traversal, and audio stability are not only infrastructure concerns.

If the audio path fails, the interview fails.

Streaming Needs Orchestration

Streaming audio, transcripts, LLM tokens, and text-to-speech audio is useful, but the product still needs rules for when to listen, when to respond, when to wait, and when to interrupt.

Transcript Handling Needs Design

Partial transcripts are useful, but they are not always stable enough for interview decisions.

Final transcripts are more reliable, but waiting too long hurts the conversation flow.

Interruption Is Cross-Layer

Interruption touches voice activity detection, playback, text-to-speech, LLM generation, and session state.

It cannot be solved only by adding a “stop audio” button.

Interview Quality Needs Structure

A realistic mock interview needs interview stages, candidate context, follow-up logic, memory, and evaluation criteria.

Without structure, the conversation can become generic.

Closing

Building an AI mock interviewer is not only about generating interview questions.

The difficult part is making the session reliable, responsive, and realistic enough for practice.

That requires a combination of realtime media engineering, streaming AI orchestration, interview-state management, and structured evaluation.

In this article, we covered the product-level architecture and some of the practical problems behind it.

In the next articles, we will go deeper into the technical layers behind this kind of system.