Modern deep neural network (DNN) models generally require a huge amount of weight and activation values to achieve good inference outcomes. Those data inevitably demand a massive off-chip memory capacity/bandwidth, and the situation gets even worse if they are represented in high-precision floating-point formats. Effort has been made for representing those data in different 8-bit floating-point formats, nevertheless, a notable accuracy loss is still unavoidable. In this paper we introduce an extremely flexible 8-bit floating-point (FFP8) format whose defining factors - the bit width of exponent/fraction field, the exponent bias, and even the presence of the sign bit - are all configurable. We also present a methodology to properly determine those factors so that the accuracy of model inference can be maximized. The foundation of this methodology is based on a key observation - both the maximum magnitude and the value distribution are quite dissimilar between weights and activations in most DNN models. Experimental results demonstrate that the proposed FFP8 format achieves an extremely low accuracy loss of $0.1\%\sim 0.3\%$ for several representative image classification models even without the need of model retraining. Besides, it is easy to turn a classical floating-point processing unit into an FFP8-compliant one, and the extra hardware cost is minor.

翻译：现代深心神经网络(DNN)模型通常要求大量重量和激活值,才能得出良好的推论结果。这些数据不可避免地要求巨大的离芯内存能力/带宽,如果以高精度浮点格式表示,情况就会更加糟糕。已经努力以不同的8位浮点格式代表这些数据,然而,显著的准确性损失仍然不可避免。在本文件中,我们引入了极灵活的8位浮点(FFP8)格式,其确定因素——Expent/fraction 字段的微宽度、Expent 偏差、甚至标志点的存在——都是不可分解的。我们还提出了一个适当确定这些因素的方法,以便模型推断的准确性可以最大化。这一方法的基础是关键观察――最大值和数值分布在大多数 DNNE模型中的重量和激活非常不相近。实验结果表明,拟议的FFFP8格式需要非常低的0.3 %的精确度损失,甚至标值比标点存在标志点,我们提出了适当确定这些因素的方法,以便模型的准确性推算出FFF8的精确度,甚至可以将一个软质再修正的模型转换为一个,而无需一个软化的硬化的软化的硬件的模型。