AssemblyAI’s Oct. 28 bench says Gemini 4 Argon led dictation cleanup, with speed and cost trade-offs varying by length.
AssemblyAI said its Voice Standard Bench still puts different LLMs in different roles for dictation cleanup, with a frontier model leading the quality score, a smaller model leading speed, and another open model leading cost at the time of the Oct. 28, 2026 update. The company said the task should be judged separately from transcription because cleanup has a deterministic target and interactive latency matters as much as output quality.
According to AssemblyAI’s Oct. 28, 2026 post, dictation cleanup in Voice Standard Bench uses a synthetic dataset with known answers and evaluates exact match, word error rate, punctuation accuracy, capitalization accuracy and formatting-command accuracy. The benchmark does not use an LLM judge for dictation, and it only tests two lengths, 30 seconds and 2 minutes, because longer dictation windows are less representative of the use case.
The company said the benchmark’s dictation prompt asks models to execute spoken commands such as “comma,” “new paragraph” and “all caps” while leaving spoken numbers spelled out as words. AssemblyAI also said the task is narrow by design: it does not score filler removal or self-corrections, even though those are part of production cleanup workflows.
On the Dictation cleanup tab as of Oct. 28, 2026, AssemblyAI said Gemini 4 Argon was the Highest Performance pick at both lengths, scoring 100 at 30 seconds and 99 at 2 minutes. 60 seconds at 2 minutes, but with lower scores of 81 and 75 respectively.
Source link
