When scaled to hundreds of billions of parameters, pretrained language models such as GPT-3 (Brown et al., 2020) achieve remarkable few-shot performance. However, enormous amounts of compute are required for training and applying such big models, resulting in a large carbon footprint and making it difficult for researchers and practitioners to use them. We show that performance similar to GPT-3 can be obtained with language models that are much "greener" in that their parameter count is several orders of magnitude smaller. This is achieved by converting textual inputs into cloze questions that contain a task description, combined with gradient-based optimization; exploiting unlabeled data gives further improvements. We identify key factors required for successful natural language understanding with small language models.

翻译：当测量到数千亿个参数时,诸如GPT-3(Brown等人,2020年)等预先培训的语言模型取得了显著的微小性能,然而,培训和应用这些大型模型需要大量计算,从而导致大量的碳足迹,研究人员和从业人员难以使用这些模型。我们证明,与GPT-3类似的语言模型可以取得类似于GPT-3的性能,因为这些语言模型的参数数小于几个数量级。这是通过将文字输入转换成含有任务描述的混凝土问题,加上基于梯度的优化;利用未贴标签的数据可以带来进一步的改进。我们确定了与小语言模型成功自然语言理解所需的关键因素。