Masked Language Modeling (MLM) has been one of the most prominent approaches for pretraining bidirectional text encoders due to its simplicity and effectiveness. One notable concern about MLM is that the special $\texttt{[MASK]}$ symbol causes a discrepancy between pretraining data and downstream data as it is present only in pretraining but not in fine-tuning. In this work, we offer a new perspective on the consequence of such a discrepancy: We demonstrate empirically and theoretically that MLM pretraining allocates some model dimensions exclusively for representing $\texttt{[MASK]}$ tokens, resulting in a representation deficiency for real tokens and limiting the pretrained model's expressiveness when it is adapted to downstream data without $\texttt{[MASK]}$ tokens. Motivated by the identified issue, we propose MAE-LM, which pretrains the Masked Autoencoder architecture with MLM where $\texttt{[MASK]}$ tokens are excluded from the encoder. Empirically, we show that MAE-LM improves the utilization of model dimensions for real token representations, and MAE-LM consistently outperforms MLM-pretrained models across different pretraining settings and model sizes when fine-tuned on the GLUE and SQuAD benchmarks.

翻译：隐蔽语言模型(MLM) 由于其简单和有效性,是预培训双向文本编码器的最突出方法之一。对MLM的一个显著关切是,美元特殊符号在培训前数据与下游数据之间造成了差异,因为它只存在于培训前,而不是微调中。在这项工作中,我们从经验上和理论上为这种差异的后果提供了一个新视角:我们展示了MLM预培训前分配一些模型维度,仅代表$textt{[MASK]$符号,导致真实符号的表示不足,并限制了在适应下游数据时预先培训模式的表达性,而没有$\textt{[MASK]}符号。我们提出MAE-LM(MAE-L(MAE-L(ML))(MAME-L(MAE-L-L(ML))(ML-MAE-L(ML(ML)))(ML-MAE-L(ML(ML-L))) (ML) (ML-ML) (ML) (ML) (ML-ML) (ML) (ML) (ML) (M) (ML-L) (ML) (ML-L) (ML) (SL) (ML) (ML) (SL) (SL) (SL) (M) (SL) (M) (SL) (M) (SL) (M) (SL) (M) (M) (M) (SL) (S) (SL) (SL) (SL) (S) (S) (S) (SL) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (SD) (S) (S) (SD) (SD) (SL) (S) (S) (SL) (S) (S) (S) (SD) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (S) (