AssemblyAI published a tutorial for building a voice agent directly on WebSockets with streaming STT, LLM Gateway and TTS.

AssemblyAI published a tutorial on Oct. 29, 2026 that walks through building a voice agent directly on raw WebSockets, without using its SDK. The post ties together streaming speech-to-text, an HTTP LLM call and text-to-speech playback, and frames the approach as a way to keep full control over every protocol frame and state transition.

The guide uses three connections and one API key: the Universal-3.6 Pro Realtime speech-to-text WebSocket, LLM Gateway over plain HTTPS, and the AssemblyAI Text-to-Speech WebSocket. According to the tutorial, the speech-to-text socket receives 16 kHz binary PCM audio and returns frames such as Begin, SpeechStarted, Turn and Termination, while the text-to-speech socket accepts Generate, Flush, Cancel, KeepAlive and Terminate and returns Audio, FlushDone, Cancelled, Termination and Error.

AssemblyAI says the example is designed for developers who want to understand or own each byte of the flow, including barge-in behavior and tool calling. The post says the sample fits in about 300 lines of Python and uses only the websocket, httpx and sounddevice libraries. It also says the same API key authenticates all three services, with LLM Gateway using an OpenAI-compatible chat-completions format.

The tutorial’s example agent is configured to answer weather questions via a get_weather tool call. In the sample code, the model request is sent with tools in the message payload, the function result is appended as a tool message, and the LLM is called again until it returns plain text.

Source link