Earables, such as True Wireless Stereo earphones and VR/AR headsets, are increasingly popular, yet their compact design poses challenges for robust voice-related applications like telecommunication and voice assistant interactions in noisy environments. Existing speech enhancement systems, reliant solely on omnidirectional microphones, struggle with ambient noise like competing speakers. To address these issues, we propose VibOmni, a lightweight, end-to-end multi-modal speech enhancement system for earables that leverages bone-conducted vibrations captured by widely available Inertial Measurement Units (IMUs). VibOmni integrates a two-branch encoder-decoder deep neural network to fuse audio and vibration features. To overcome the scarcity of paired audio-vibration datasets, we introduce a novel data augmentation technique that models Bone Conduction Functions (BCFs) from limited recordings, enabling synthetic vibration data generation with only 4.5% spectrogram similarity error. Additionally, a multi-modal SNR estimator facilitates continual learning and adaptive inference, optimizing performance in dynamic, noisy settings without on-device back-propagation. Evaluated on real-world datasets from 32 volunteers with different devices, VibOmni achieves up to 21% improvement in Perceptual Evaluation of Speech Quality (PESQ), 26% in Signal-to-Noise Ratio (SNR), and about 40% WER reduction with much less latency on mobile devices. A user study with 35 participants showed 87% preferred VibOmni over baselines, demonstrating its effectiveness for deployment in diverse acoustic environments.

翻译：耳戴设备，如真无线立体声耳机和VR/AR头显，正日益普及，但其紧凑的设计为嘈杂环境下的语音通信和语音助手交互等鲁棒性语音应用带来了挑战。现有的语音增强系统仅依赖全向麦克风，难以应对如竞争说话者等环境噪声。为解决这些问题，我们提出了VibOmni，一种轻量级、端到端多模态语音增强系统，用于耳戴设备，利用广泛可用的惯性测量单元（IMU）捕获的骨传导振动。VibOmni集成了一个双分支编码器-解码器深度神经网络，以融合音频和振动特征。为克服配对音频-振动数据集的稀缺性，我们引入了一种新颖的数据增强技术，从有限录音中建模骨传导函数（BCF），仅需4.5%的频谱图相似度误差即可生成合成振动数据。此外，多模态信噪比估计器支持持续学习和自适应推理，在动态嘈杂环境中优化性能，无需设备端反向传播。在来自32名志愿者使用不同设备的真实世界数据集上评估，VibOmni在语音质量感知评估（PESQ）上实现了高达21%的提升，信噪比（SNR）提升26%，词错误率（WER）降低约40%，且在移动设备上延迟显著减少。一项涉及35名参与者的用户研究表明，87%的参与者更倾向于VibOmni而非基线方法，证明了其在多样化声学环境中部署的有效性。