Deploying Transformer models on edge devices is limited by latency and energy budgets. While INT8 quantization effectively accelerates the primary matrix multiplications, it exposes the softmax as the dominant bottleneck. This stage incurs a costly dequantize-softmax-requantize detour, which can account for up to 65% of total attention latency and disrupts the end-to-end integer dataflow critical for edge hardware efficiency. To address this limitation, we present IntAttention, the first fully integer, plug-and-play attention pipeline without retraining. At the core of our approach lies IndexSoftmax, a hardware-friendly operator that replaces floating-point exponentials entirely within the integer domain. IntAttention integrates sparsity-aware clipping, a 32-entry lookup-table approximation, and direct integer normalization, thereby eliminating all datatype conversion overhead. We evaluate IntAttention and demonstrate consistent and substantial gains. Our method achieves up to 3.7x speedup and 61% energy reduction over FP16 baselines and 2.0x faster than conventional INT8 attention pipelines on Armv8 CPUs. These gains are achieved with high-fidelity accuracy comparable to baselines across diverse language and vision models, enabling practical and efficient Transformer inference on commodity edge devices. Code will be released in later version of this work.

翻译：在边缘设备上部署Transformer模型受到延迟和能耗预算的限制。虽然INT8量化有效加速了主要的矩阵乘法运算，但softmax层却暴露为主要的性能瓶颈。该阶段涉及昂贵的去量化-softmax-重量化迂回操作，可占据注意力总延迟的65%，并破坏了边缘硬件效率所依赖的端到端整数数据流。为突破这一限制，本文提出IntAttention——首个无需重新训练、即插即用的全整数注意力流水线。我们方法的核心是IndexSoftmax，这是一种硬件友好的算子，可在整数域内完全替代浮点指数运算。IntAttention集成了稀疏感知裁剪、32项查找表近似和直接整数归一化技术，从而消除了所有数据类型转换开销。我们对IntAttention进行评估并展示了持续且显著的性能提升。该方法在Armv8 CPU上相比FP16基线实现了最高3.7倍加速和61%能耗降低，比传统INT8注意力流水线快2.0倍。这些增益是在保持与基线相当的高保真度精度下实现的，涵盖多种语言和视觉模型，为商用边缘设备上实现实用高效的Transformer推理提供了可能。代码将在本工作的后续版本中发布。