Code-switching (CS) speech translation (ST) refers to translating speech that alternates between two or more languages into a target language text, which poses significant challenges due to the complexity of semantic modeling and the scarcity of CS data. Previous studies tend to rely on the model itself to implicitly learn semantic modeling during training, and resort to inefficient and costly manual annotations for these two challenges. To mitigate these limitations, we propose enhancing Large Language Models (LLMs) with a Mixture of Experts (MoE) speech projector, where each expert specializes in the semantic subspace of a specific language, enabling fine-grained modeling of speech features. Additionally, we introduce a multi-stage training paradigm that utilizes readily available monolingual automatic speech recognition (ASR) and monolingual ST data, facilitating speech-text alignment and improving translation capabilities. During training, we leverage a combination of language-specific loss and intra-group load balancing loss to guide the MoE speech projector in efficiently allocating tokens to the appropriate experts, across expert groups and within each group, respectively. To bridge the data gap across different training stages and improve adaptation to the CS scenario, we further employ a transition loss, enabling smooth transitions of data between stages, to effectively address the scarcity of high-quality CS speech translation data. Extensive experiments on widely used datasets demonstrate the effectiveness and generality of our approach.

翻译：语码转换（CS）语音翻译（ST）是指将两种或多种语言交替出现的语音翻译成目标语言文本，由于语义建模的复杂性以及CS数据的稀缺性，该任务面临显著挑战。先前的研究倾向于依赖模型本身在训练过程中隐式学习语义建模，并针对这两项挑战采用低效且成本高昂的人工标注方法。为缓解这些局限，我们提出通过集成专家混合（MoE）语音投影器来增强大型语言模型（LLMs），其中每个专家专注于特定语言的语义子空间，从而实现对语音特征的细粒度建模。此外，我们引入一种多阶段训练范式，利用易于获取的单语自动语音识别（ASR）和单语ST数据，促进语音-文本对齐并提升翻译能力。在训练过程中，我们结合使用语言特定损失和组内负载均衡损失，分别指导MoE语音投影器在专家组之间以及各组内部分配令牌至合适的专家。为弥合不同训练阶段间的数据差距并提升对CS场景的适应性，我们进一步采用过渡损失，实现数据在阶段间的平滑过渡，以有效应对高质量CS语音翻译数据稀缺的问题。在广泛使用的数据集上进行的大量实验验证了我们方法的有效性和普适性。