Advances in deep learning have led to state-of-the-art performance across a multitude of speech recognition tasks. Nevertheless, the widespread deployment of deep neural networks for on-device speech recognition remains a challenge, particularly in edge scenarios where the memory and computing resources are highly constrained (e.g., low-power embedded devices) or where the memory and computing budget dedicated to speech recognition is low (e.g., mobile devices performing numerous tasks besides speech recognition). In this study, we introduce the concept of attention condensers for building low-footprint, highly-efficient deep neural networks for on-device speech recognition on the edge. An attention condenser is a self-attention mechanism that learns and produces a condensed embedding characterizing joint local and cross-channel activation relationships, and performs selective attention accordingly. To illustrate its efficacy, we introduce TinySpeech, low-precision deep neural networks comprising largely of attention condensers tailored for on-device speech recognition using a machine-driven design exploration strategy, with one tailored specifically with microcontroller operation constraints. Experimental results on the Google Speech Commands benchmark dataset for limited-vocabulary speech recognition showed that TinySpeech networks achieved significantly lower architectural complexity (as much as $507\times$ fewer parameters), lower computational complexity (as much as $48\times$ fewer multiply-add operations), and lower storage requirements (as much as $2028\times$ lower weight memory requirements) when compared to previous work. These results not only demonstrate the efficacy of attention condensers for building highly efficient networks for on-device speech recognition, but also illuminate its potential for accelerating deep learning on the edge and empowering TinyML applications.

翻译：深层学习的进展导致在众多语音识别任务中取得了最先进的表现。然而,广泛部署深神经网络以在边缘安装高脚印、高效率的深神经网络以在边缘进行语音识别仍然是一个挑战,特别是在记忆和计算资源高度受限的边缘情景中(例如,低功率嵌入装置),或者专用于语音识别的记忆和计算预算较低(例如,移动设备履行语音识别以外的许多任务)。在本研究中,我们引入了为在边缘建立低脚印、高效率的深神经网络以在边缘安装高设备语音识别而安装高效率的深神经网络的概念。 关注冷凝器是一种自我关注机制,它学习并产生精精精缩的嵌入式嵌入式本地和跨通道激活关系(例如,低功率嵌入装置),为了说明其功效,我们引入了TinySpeech、低精锐度的深神经网络,主要是通过机械驱动设计开发的语音识别分解器进行现场识别,但有高度定制的微控制操作限制。谷语音语音指令的实验性结果,而不是自我定位服务器上较低精密的精度网络,也显示低精度的精度的精度的精度的精度的精度的精度,因为其精度的精度的精度的精度的精度的精度的精度,因此的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度,因此的精度的精度的精度的精度的精度的精度,因此的精度的精度的精度的精度,因此的精度的精度的精度也的精度,因此的精度的精度的精度的精度的精度的精度的精度,因此的精度的精度的精度的精度的精度也的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度的精度,