Creating an essay based on a few given topics is a challenging NLP task. Although several effective methods for this problem, topic-to-essay generation, have appeared recently, there is still much room for improvement, especially in terms of the coverage of the given topics and the coherence of the generated text. In this paper, we propose a novel approach called TegFormer which utilizes the Transformer architecture where the encoder is enriched with domain-specific contexts while the decoder is enhanced by a large-scale pre-trained language model. Specifically, a \emph{Topic-Extension} layer capturing the interaction between the given topics and their domain-specific contexts is plugged into the encoder. Since the given topics are usually concise and sparse, such an additional layer can bring more topic-related semantics in to facilitate the subsequent natural language generation. Moreover, an \emph{Embedding-Fusion} module that combines the domain-specific word embeddings learnt from the given corpus and the general-purpose word embeddings provided by a GPT-2 model pre-trained on massive text data is integrated into the decoder. Since GPT-2 is at a much larger scale, it contains a lot more implicit linguistic knowledge which would help the decoder to produce more grammatical and readable text. Extensive experiments have shown that the pieces of text generated by TegFormer have better topic coverage and higher text coherence than those from SOTA topic-to-essay techniques, according to automatic and human evaluations. As revealed by ablation studies, both the Topic-Extension layer and the Embedding-Fusion module contribute substantially to TegFormer's performance advantage.

翻译：在几个特定主题的基础上创建文章是一项具有挑战性的 NLP 任务。 尽管最近出现了解决这一问题的若干有效方法, 即主题到分析的生成, 但仍有很大的改进空间, 特别是在特定主题的覆盖范围和生成文本的一致性方面。 在本文中, 我们提议了一种叫TegFormer 的新颖方法, 使用变异器结构, 使编码器与特定域背景相丰富, 而解码器则通过一个大型的预培训语言模型强化。 具体地说, 一个Exph{ Topic-Extension} 层, 捕捉到特定主题及其特定技术之间的更高互动, 并插入了它们各自的域- 技术在编码中。 由于给给定的主题通常是简洁和稀少的, 这样增加的层可以带来更多与主题相关的语义定义, 此外, 一个将特定域的词嵌入模块, 从给给给给给给给给定的版本和GPTF 提供的普通词嵌入。 由于GP-2 预培训前的文本将更精细的解到解的解的版本, 将产生更细的解的解的解的文本到解式的解式的解式的版本。