Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM's output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.

翻译：受过训练的以变异器为基础的语言模型(LMs) 展示出非凡的自然语言生成能力。 具有巨大的潜力, 控制这种LMs的文本生成正在引起注意。 虽然有些研究试图控制生成文本的高级属性( 如情绪和主题), 但是在文字和语句层面仍然缺乏对其内容的更精确控制。 在这里, 我们提议内容- 调控器(Cocon) 来控制含有内容输入的LM 输出文本, 在精细的层次上。 在我们的自我监督方法中, COCon块学会帮助LM 完成一个部分观测到的文本序列, 通过对LM 中保留的内容输入进行调节 。 我们通过实验, COCon 可以自然地将目标内容纳入生成文本中, 并以零速的方式控制高文本属性 。