AssemblyAI's benchmark found small models close to larger ones on short calls, but less reliable on longer meetings and harder extraction tasks.

AssemblyAI said its benchmark data shows small language models can match larger systems closely on short voice transcripts, but lose that edge as transcript length increases. In its Voice Standard Bench run published Oct. 13, 2026, the company said the performance gap widens on longer meetings, while the cost advantage of smaller models narrows once the task becomes more complex.

The post, published Oct. 20, 2026, frames the question as one of transcript length rather than model class alone. AssemblyAI said the current Voice Standard Bench run compares models on real speech-to-text output at about 30 seconds, 2 minutes, 15 minutes and 60 minutes. In the chart it provided, GPT-6 Luna scored 96, 94, 89 and 88 across those lengths, while gemma-4-31b scored 95, 92, 87 and 85, Qwen3.5 4B Fast scored 93, 89, 64 and 63, and another model listed in the post scored 74, 64, 47 and 31. AssemblyAI said scores within 2 to 3 points are effectively tied, and that gemma-4-31b stays close to the leader on short inputs before dropping more sharply by 15 minutes.

The company also said the economics change with length. In the current run, the top-scoring model costs 15 to 77 times more per hour of audio than a model within two quality points of it, according to AssemblyAI. On hour-long transcripts, that premium narrows to about 4 times, which the company said is where larger models begin to earn their higher cost.

Source link