We investigate in this paper how distributions of occupations with respect to gender is reflected in pre-trained language models. Such distributions are not always aligned to normative ideals, nor do they necessarily reflect a descriptive assessment of reality. In this paper, we introduce an approach for measuring to what degree pre-trained language models are aligned to normative and descriptive occupational distributions. To this end, we use official demographic information about gender--occupation distributions provided by the national statistics agencies of France, Norway, United Kingdom, and the United States. We manually generate template-based sentences combining gendered pronouns and nouns with occupations, and subsequently probe a selection of ten language models covering the English, French, and Norwegian languages. The scoring system we introduce in this work is language independent, and can be used on any combination of template-based sentences, occupations, and languages. The approach could also be extended to other dimensions of national census data and other demographic variables.

翻译：我们在本文中研究了职业领域中性别分布在预先训练的语言模型中的反映。这些分布并不总是符合规范，也不一定反映现实的描述性评估。在本文中，我们介绍了一种衡量预先训练的语言模型在规范和描述性职业分布方面程度的方法。为此，我们使用了法国、挪威、英国和美国国家统计机构提供的关于性别-职业分布的官方人口统计信息。我们手动生成了结合了带性别的代词和名词以及职业的基于模板的句子，然后检查了涵盖英语、法语和挪威语的十种语言模型的选择。我们在本文中引入的评分系统是与语言无关的，可以用于任何基于模板的句子、职业和语言的组合。这种方法也可以扩展到国家人口普查数据的其他维度和其他人口统计变量。