Every conversation carries two layers of information: what was said and who said it. Speech recognition captures and transcribes the words. Speaker diarization classifies who spoke when, helping applications connect what was said to the right participant.

Consider a transcript from a meeting, customer call, or podcast in which every sentence is correct but none is attributed to a speaker. You can read the words, but you cannot reliably tell who made a commitment, who raised an objection, or which participant interrupted. Search, summaries, action items, conversation analytics, and voice-agent memory all become less useful.

Speaker diarization identifies the time intervals during which each speaker is active, including intervals where people talk over one another. Those speaker timestamps can then be combined with automatic speech recognition (ASR) to create a speaker-attributed transcript.

NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on VoiceArena's Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). Supporting up to eight speakers across live and recorded conversations, it handles overlapping speech, chunked processing for flexible recording lengths, and customizable streaming latency.

Earlier models like NVIDIA Streaming Sortformer established this approach for four-speaker diarization, including the streaming diar_streaming_sortformer_4spk-v2.1 checkpoint used as the baseline below. Nemotron 3 Diarization expands support to eight speakers and improves the accuracy and throughput measured in the following evaluations.

Diarization systems must solve two related problems. First, they must detect speech and assign it to the correct speaker. Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between a speaker's turns.

Source link