The industry is moving from high-precision formats (FP32, FP16, BF16) to lower-precision data types for Machine Learning (ML) inference and training. In cloud and data center environments, lower-precision data types deliver higher throughput by packing more compute per cycle from the same platform. On edge and mobile devices, smaller formats reduce model size, memory footprint and bandwidth, which directly affects power consumption and latency.

Arm, Intel, and NVIDIA published an OCP 8-bit Floating Point Specification (OFP8). It is an interchange format that allows software systems to share the models easily. FP8 is not a future concept, it is being deployed today. It is already supported by NVIDIA’s Hopper H100 GPU, with software deployment appearing in major ML frameworks.

Arm added the FP8 support to the Arm v9 ISA and is exploring the best way to integrate the FP8 across all the ML platforms.

This blog post explains the FP8 data formats, the new Floating-point Mode Register (FPMR), and the instruction set that makes FP8 practical on Arm CPUs.

Unlike IEEE FP64, FP32, and FP16, FP8 is not a single format, it is a family. The whitepaper FP8 Formats for Deep Learning proposes an FP8 format consisting of two encodings: E4M3 and E5M2.

With only 8 bits available, FP8 makes a trade-off between precision and dynamic range. A single encoding cannot fit all tensors equally well. Activations and weights usually have a smaller and more stable value distribution after training or calibration. They can therefore benefit from more mantissa bits, which help represent nearby values more precisely.

Source link