AMD reports up to 58.1% latency gains in BERT inference using ZenDNN on EPYC processors, showcasing significant performance improvements.
AMD has unveiled performance benchmarks for low-latency CPU inference on its EPYC processors, leveraging optimizations in its ZenDNN library and zentorch PyTorch plugin. The tests, conducted in September 2026, focused on BERT-Base uncased inference using Hugging Face's implementation. The results highlight significant latency gains, particularly when using BF16 precision, with improvements reaching up to 58.1% compared to ONNX Runtime FP32.
AMD's testing spanned two configurations: a 96-core EPYC 9654 bare-metal system with NPS4 and an 80-core EPYC 9V74 virtual machine with NPS5. Across 42 configurations, zentorch outperformed ONNX Runtime in 83% of cases, tied in 10%, and trailed in 7% on the bare-metal setup. In the virtual machine, zentorch achieved similar results, leading in 83%, tying in 7%, and trailing in 10% of configurations.
For FP32 precision, the average latency gain was 9.7% on bare metal and 9.1% in the virtual machine. BF16 precision delivered more substantial improvements, with average gains of 47.3% and 46.1%, respectively. The highest reported latency gain occurred in the BF16 configuration, achieving 58.1% at four instances, sequence length 128, and 16 cores.
AMD attributed the performance improvements to several key optimizations in ZenDNN and zentorch:
These interventions narrowed the multi-instance performance deficit at shorter sequence lengths. For example, at sequence length 32, the deficit was reduced from 21% to 7% below ONNX Runtime.
The benchmarks demonstrate AMD's focus on optimizing CPU-based inference for AI workloads such as search ranking, classification, and retrieval-augmented generation reranking.
Source link
