Transformers in their common form are inherently limited to operate on whole token sequences rather than on one token at a time. Consequently, their use during online inference on time-series data entails considerable redundancy due to the overlap in successive token sequences. In this work, we propose novel formulations of the Scaled Dot-Product Attention, which enable Transformers to perform efficient online token-by-token inference on a continual input stream. Importantly, our modifications are purely to the order of computations, while the outputs and learned weights are identical to those of the original Transformer Encoder. We validate our Continual Transformer Encoder with experiments on the THUMOS14, TVSeries and GTZAN datasets with remarkable results: Our Continual one- and two-block architectures reduce the floating point operations per prediction by up to 63x and 2.6x, respectively, while retaining predictive performance.

翻译：通用式的变换器在本质上限于按全等序列运行,而不是一次以一个符号运行。 因此,在时间序列数据的在线推断中,由于连续的代号序列重叠,在时间序列数据的在线推断中使用这些变换器需要大量冗余。 在这项工作中,我们提议了缩放点-Producle 注意的新配方,使变换器能够在连续输入流上高效的在线逐个象征性推论。 重要的是,我们的修改完全按照计算顺序进行,而产出和学到的重量与原变换器的相同。 我们用THUMOS14、TeVSeries和GTZAN数据集的实验来验证我们的连续变换转换器,结果显著:我们的连续一和两块结构将每个预测的浮点操作分别减少63x和2.6x,同时保留预测性能。