AssemblyAI published a tutorial for building a server-side Agora voice agent using Universal-3.6 Pro Realtime, LLM Gateway and Text-to-Speech.

AssemblyAI has published a technical tutorial showing how to build a server-side voice agent for an Agora RTC channel using its speech-to-text, LLM Gateway and text-to-speech stack. The guide is centered on a Python bot that joins an Agora call like any other participant, listens to the caller and speaks replies back into the channel, with barge-in handled in the application loop.

In the tutorial published on 29 Oct 2026, AssemblyAI says the bot combines Agora's WebRTC transport with three AssemblyAI layers on one API key: Universal-3.6 Pro Realtime for streaming speech-to-text, LLM Gateway for model responses, and AssemblyAI Text-to-Speech for the spoken reply. The company says the architecture keeps audio routing on Agora while the conversation loop stays in user code.

The example uses 16 kHz, 16-bit, mono PCM end to end. According to the guide, Agora delivers the caller audio at 16 kHz, Universal-3.6 Pro Realtime accepts that rate directly, and the text-to-speech session is opened with sample_rate=16000 and encoding="pcm_s16le", so the bot does not need resampling code. AssemblyAI also says the tutorial is written against agora_python_server_sdk 2.4.10 and requires Python 3.10+.

19% mean word error rate on AssemblyAI's English voice-agent benchmark and a median 307 ms to final transcript on the Pipecat open benchmark, both as company-reported results. The guide also describes barge-in logic in which a real caller interruption cancels synthesis in progress and clears buffered audio in Agora, while short backchannels such as "mhm" or "yeah" are ignored.

Source link