Large multilingual language models typically share their parameters across all languages, which enables cross-lingual task transfer, but learning can also be hindered when training updates from different languages are in conflict. In this paper, we propose novel methods for using language-specific subnetworks, which control cross-lingual parameter sharing, to reduce conflicts and increase positive transfer during fine-tuning. We introduce dynamic subnetworks, which are jointly updated with the model, and we combine our methods with meta-learning, an established, but complementary, technique for improving cross-lingual transfer. Finally, we provide extensive analyses of how each of our methods affects the models.

翻译：大型多语文模式通常在所有语文中共享其参数,这可以实现跨语言任务转移,但是,在来自不同语言的培训更新出现冲突时,学习也会受到阻碍。在本文件中,我们提出了使用具体语言子网络的新方法,这些网络控制跨语言参数共享,以减少冲突,并在微调期间增加积极的转移。我们引入动态子网络,这些子网络与该模式共同更新,我们将我们的方法与元学习相结合,这是改进跨语言转让的既定但互补的技术。最后,我们提供了对我们每种方法如何影响模式的广泛分析。