Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.

翻译：尽管时间变数很重要,但在国家劳工政策及语言示范文献中,时间变数在很大程度上被忽略了。在本文件中,我们介绍了“时间LM”,这是一套专门处理日新月异Twitter数据的语文模型。我们表明,持续学习战略有助于提高基于推特的语言模型处理未来和分配外推特的能力,同时使其具有标准化和更加单一的基准的竞争力。我们还进行了一些定性分析,表明它们如何应对特定实体或概念漂移活动的趋势和高峰。