Initial Data Formats (6), previous AI acceleration operators (5) are quite numerous, so I won’t release them for now. (INT8、Bfloat、FP8、MSFP、MX）
If there is one factor that has driven the significant improvement in the efficiency of artificial intelligence (AI) over the past decade, it is the evolution of data formats. The industry has invested heavily in researching data formats that can balance precision, range, and memory/computational efficiency. While traditional formats such as integers, fixed-point numbers, and IEEE floating-point numbers have long been the pillars of digital computation, AI has driven the development of specialized formats that define their characteristics.
This article first introduces the basics: integers, fixed-point, and IEEE floating-point representations,
Then it delves into innovations in the AI field, such as Google Brain floating-point (BFloat16), FP8, Microsoft floating-point (MSFP), and micro-exponent (MX) formats, exploring their precision, scaling, and memory usage.
1) Integers (int):
INT is short for Integer, which is an integer type. Integers are like the numbers used for counting, such as 1、2、3…… with no decimal part.The integer system is a simple and efficient representation method for integers. The integer system is discretely uniformly distributed, with uniform spacing between points.
INT4 represents an integer using 4 bits of binary, for example, INT4 can represent a limited range of integers because 4 bits of binary can represent at most16 different numbers; if it is a signed number, the range is usually from -8 to 7; if it is an unsigned number, the range is from 0 to 15.
Similarly, INT8 represents integers using 8 bits of binary. The signed range of INT8 is from -128 to 127, and the unsigned range is from 0 to 255. For example, INT8 (8-bit integer) represents integers between -128 and 127.
Disadvantages:
Integers have limitations in precision and range; pure integer systems cannot represent decimals (precision). Additionally, integers themselves do not have a scaling factor to adjust the range. Pure integer representation is not common in AI.
2) Fixed Point:
In addition to integer values, fixed-point numbers introduce a decimal point (a fixed scaling factor), for example, stored as123, multiplied by a fixed scaling factor of0.001 , results in0.123. Fixed-point INT8 is commonly used in AI deployment (inference), processed through quantization after training. At this point, the model parameters are converted toINT8, reducing memory usage and speeding up computation without retraining the entire model. Its distribution is also discretely uniform.
Disadvantages:
The dynamic range of fixed-point numbers is limited: with a fixed scaling factor, if both large and small numbers need to be represented, many bits are required, leading to significant memory usage.
For example, if the fixed scaling factor is0.001 , the smallest positive non-zero value it can represent is 0.001 (1 * 0.001). However, to represent 100, it would need to store 100,000 (because 100 = 0.001 * 100,000), which would require about17 bits; this is more than double the number of bits required to represent 100 in an integer system.
3) Floating Point (IEEE754-2019):
To address the scaling issue without increasing bit width, floating-point numbers introduce a dynamic scaling factor through exponents. For example, the 16-bit floating-point number (FP16) compliant with IEEE standards consists of 1 sign bit, 5 exponent bits, and 10 mantissa bits.
Floating-point representation:
1) If Exp>0 (i.e., normalized number),
value=(-1)sign x (1+Mantissa) x 2Exp-bias
2) If Exp=0 (i.e., denormalized number),
value=(-1)sign x (0+Mantissa) x 2Exp-bias
Where,
l Signsign: Determines positive (0) or negative (1)
l ExponentExp:Exponent represents the scaling factor, using offset encoding (actual exponent= encoded valueExp-bias).
n Bias:1 constant (in FP16, bias=15), allowing the exponent to represent both positive and negative values while storing them as unsigned integers.
The stored5 bits of exponent can represent0-31; however, 0 and 31 ( all 0 or all 1) are reserved for special numbers. Therefore, the effective positive value range of the scaling factor (1-30) is from0.00061 – 32,768.
2(1-bias) = 2(-14) ≈ 0.000061
2(30-bias) = 2(15) = 32768
l Mantissa: also known asfractionfraction, provides the precision of the decimal part(binary representation).
If it is a normalized number, the number is normalized, meaning there is an implicit leading1 leading 1.0 (not stored).
Besides the leading 1 , 10 bits of mantissa are stored, with a precision of 11 bits, where the effective range is 1 ~ (2-2^(-10))
In IEEE FP16, the minimum normal effective value
In IEEE FP16, the maximum normal effective value
Considering effective bits and exponent fields, the maximum normal value representable by FP16 is65504, while the minimum positive non-zero normal value is0.000061.
Any exponent-based floating-point system has a non-linear distribution and has denser representations near0. This is because the scaling factor grows logarithmically (2^exponent); when one exponent becomes the next, the step size between two consecutive numbers doubles, while the mantissa provides uniform precision within each exponent range..
Note: At each exponent, scan 10 effective digits. As the exponent transitions from one to the next, the interval between values changes, doubling the effective digits. Thus, denser representations can be obtained near 0.
Supplementary representation of all types of FP:
|
Type |
Exponent Bits |
Mantissa Bits |
Example |
|
±0 |
All0 |
All0 |
1.0 / +0.0 = +∞ |
|
**±∞** |
All1 |
All0 |
1.0 / 0.0 = +∞ |
|
NaN |
All1 |
Non-all0 |
0/0、∞ – ∞ |
|
Normalized |
Non-all0/All1 |
Any |
Standard range values (e.g., 3.14） |
|
DenormalizedNumber |
All0 |
Non-all0 |
Decimal close to0 (e.g., 1.4e-45） |
Note:
1）Overflow occurs when the result is too large to be accurately represented, which can be rounded to return±∞.
2）Underflow occurs when the result is too small, exceeding the normal range, which can attempt to use denormalized numbers for representation.Denormalized numbers fill the gap between ±0 and the smallest normal number, avoiding sudden underflow to zero.
MultipleNaN meanings:
|
Type |
Full Name |
Behavior Characteristics |
MantissaHighest Bit |
Usage |
|
NaN |
Not a Number |
Indicates an invalid or undefined operation result (e.g.,0/0,√(-1)), comparing with any value (including itself) returns false |
No strong requirement |
Identifies illegal computation results |
|
qNaN |
Quiet NaN |
Silently processed: When participating in operations, it does not trigger exceptions, and the result remains NaN (e.g., NaN + 1 = NaN) |
1 |
Tolerates invalid values, avoiding program interruption |
|
sNaN |
Signaling NaN |
Signals trigger: When used, it immediately raises an exception (e.g., hardware interrupt), used for debugging and strict error detection |
0 (and mantissa non-zero) |
Debugging scenarios, capturing illegal operations |
Disadvantages:
Although floating-point has extremely high precision, it has limitations in range; very small/large gradient updates during AI training, as well as the accumulation of large products in matrix multiplication during AI training/inference, require a wider dynamic range and can tolerate reduced precision. Allocating more bits for the decimal part may lead to underflow (very small numbers rounded to 0) or overflow (truncated to the maximum representable value), or values must be scaled (software overhead) to the representable range.
4) BFloat:
In 2019, Google released Brain Float (Bfloat) as a custom data format for AI training designed specifically for TPU. The concept is simple:(FP16:1+5+10 -> BF16:1+8+7) recognizing that gradients often prioritize range over precision, increasing the number of bits for the exponent and reducing the number of bits for the mantissa. For example, compared to FP16 with5 bits, BF16 uses8 bits for the exponent, thus providing the range of FP32 while maintaining compact16 bits of memory usage.
FP 16 vs Bfloat 16 Data Format Comparison
Disadvantages:
Although its dynamic range is wide, its precision is low (equivalent to only about 3 decimal places), so when applied to small errors in input values, it may significantly affect the output function/layer (e.g., exponential function, logarithmic function, S-shaped function). Additionally, in large-scale AI deployments (inference), it still occupies more memory compared to fixed-point INT8.
5) FP8:
FP8 was jointly launched by Nvidia, Intel, and ARM in 2022, representing a natural evolution of FP16/bfloat16, aimed at achieving more efficient data storage and computational acceleration, and is widely used and natively supported in AI inference accelerators. Furthermore, using the same data, FP8 training does not require post-training quantization, significantly simplifying FP8 inference, which typically requires quantization when deploying networks trained with 32-bit or 16-bit floating-point formats for INT8 inference. FP8 was standardized by the Open Compute Project (OCP) in 2023.
FP8 uses two formats:
lE4M3 (4 exponent bits, 3 mantissa bits, also known as FP8P): prioritizes precision but has a smaller range. Suitable for nonlinear regularization and activation functions, where small errors in input values can lead to significant changes in output. Recommended for weights and activation functions in forward propagation/AI inference.
lE5M2 (5 exponent bits, 2 mantissa bits, also known as FP8R): prioritizes range, with slightly lower precision. More suitable for gradients that can exhibit a larger dynamic range during backpropagation. Its design is similar to a simplified version of IEEE FP16, making conversion between the two more convenient.
Comparison of numerical ranges for FP8 formats e4m3 and e5m2
Note: FP8 e4m3 (green) shows a smaller range with finer granularity; while FP8 e5m2 (red) shows a larger range with coarser granularity. These numbers are generated by covering all possible exponents of each format and periodically sampling normalized effective digits. The scatter plot provides a representative visualization of the numerical distribution.
Disadvantages:
In networks where FP8 precision is insufficient, computations are still performed at higher precision (e.g., FP16, bfloat16) and then converted back to FP8 for storage. Similarly, networks requiring a larger dynamic range than FP8 provide need software expansion (overhead).
6) Microsoft Floating Point MSFP16:
Microsoft developed this technology in 2020, aimed at achieving low-cost large-scale inference (Brainwave project), eliminating the need for manual scaling calibration required for fixed-point INT8 or relatively expensive FP16/Bfloat16 formats. The concept is very clever: MSFP does not assign a unique exponent to each element like floating-point numbers, nor does it assign a shared exponent to all elements like fixed-point numbers, but rather assigns a shared exponent for every n elements (bounding box size). This shared exponent (8 bits) is usually the maximum value within the bounding box, representing the range of values, helping to minimize the impact of outliers on the elements within the bounding box.
Comparison of Microsoft Floating Point (MSFP) with IEEE Floating Point (FP16) format.
Note: MSFP assigns a shared exponent for every n elements, while IEEE FP assigns a private exponent for each element.
Example of converting IEEE FP16 to MSFP16:
Assuming a bounding box consisting of 4 FP16 values [50,25,12.5,3.125] represented in MSFP16.
Step 1: Determine the shared exponent (for easier understanding, using decimal representation).
Step 2: Adjust the mantissa (each mantissa is shifted right based on the difference between the shared exponent and the individual value’s exponent):
Result: These values are now stored as:
lShared Exponent:5
lMantissa:[1.5625 ,0.781, 0.39, 0.098].
lThus, it reduces storage by about 50% of the amount.
Disadvantages:
Although this method can significantly improve memory efficiency, if the range of values of the n elements in the bounding box is wide (i.e., very small and very large), the precision of small elements in the box will be significantly reduced due to all mantissas in the box being scaled by the maximum exponent. Additionally, this method is proprietary to Microsoft.
7) MX::
OCP released the MX specification v1.0 in 2023, which is considered the first open standard for shared exponent data types. Although the idea of sharing one exponent for each block of elements was initially implemented in MSFP, MX is the first industry standard aimed at artificial intelligence. The main difference between MX and MSFP is that, in addition to the shared exponent, each FP element in the block can also retain its own private exponent, thus addressing the precision loss issue of MSFP through greater storage capacity. MX is gaining support from AI accelerators/GPUs, achieving a good balance between precision, range, and hardware efficiency.
Data format of MX:
Figure 5.1 – Data format of MX
It can be divided based on the type of element: MX’s FP and INT type data (i.e., MXFP and MXINT).
Table 1.1 – Format names and parameters of mx compatible formats
1）Share Scale : M8E0
Actual exponent range= Scale – bias = (0~254) – 127 = (-127 ~127)
All1 indicatesNaN
Note:K=32, that is, 32 elements share1 scale value
2）FP（floating point）
S + Exponent(bias) + Mantissa
FP8(E4M3) : 1 + 4(7) + 3
FP8(E5M2) : 1 + 5(15) + 2
FP6(E2M3) : 1 + 2(1) + 3
FP6(E3M2) : 1 + 3(3) + 2
FP4(E2M1) : 1 + 2(1) + 1
3)INT（fixed point）
Int 8 encoding format:1 bit sign bit, 1 bit integer bit and6 bits mantissa bits
Implicit scale: 2^(-6)
Interesting fact: MX was originally proposed by Microsoft and Meta at the International Symposium on Computer Architecture (ISCA) in 2023. The exponent sharing mechanism in the original MX differs from the later standardized mechanism by OCP. In the initial version, one block (usually containing 16 elements) was further subdivided into smaller groups of 2 elements, each sharing 1 bit exponent in addition to the block-level shared exponent.
Comparison of MX first introduced at ISCA 2023 with OCP standardized MX
Summary:
Data formats essentially involve trade-offs between precision, range, and hardware (computation/storage/transmission) efficiency. There is a wide variety of data formats, presenting enormous exploration space. Mixed precision AI inference and training have become popular and achieved better performance.
S + Exponent(bias) + Mantissa
FP32(E8M23) : 1 + 8(127) + 23
TF32(E8M10) : 1 + 8(127) + 10
FP16(E5M10) : 1 + 5(15) + 10
BF16(E8M7) : 1 + 8(127) + 7
BF8(E3M4) : 1 + 3(3) + 4
FP8(E4M3) : 1 + 4(7) + 3
FP8(E5M2) : 1 + 5(15) + 2
FP6(E2M3) : 1 + 2(1) + 3
FP6(E3M2) : 1 + 3(3) + 2
FP4(E2M1) : 1 + 2(1) + 1
Int 8 :Implicit scale: 2^(-6)
Int 4 :Implicit scale: 2^(-2)
References:
1）Even beginners can understand!INT4、INT8、FP8、FP16、FP32 quantization
https://www.53ai.com/news/LargeLanguageModel/2025022275609.html
2）mx python implementation
microsoft/microxcaling: PyTorch emulation library for Microscaling (MX)-compatible data formats