In principle, applying variational autoencoders (VAEs) to sequential data offers a method for controlled sequence generation, manipulation, and structured representation learning. However, training sequence VAEs is challenging: autoregressive decoders can often explain the data without utilizing the latent space, known as posterior collapse. To mitigate this, state-of-the-art models weaken the powerful decoder by applying uniformly random dropout to the decoder input. We show theoretically that this removes pointwise mutual information provided by the decoder input, which is compensated for by utilizing the latent space. We then propose an adversarial training strategy to achieve information-based stochastic dropout. Compared to uniform dropout on standard text benchmark datasets, our targeted approach increases both sequence modeling performance and the information captured in the latent space.

翻译：原则上,对顺序数据应用变式自动解码器(VAEs)为控制序列生成、操纵和结构化代表性学习提供了一种方法。然而,培训序列VAEs具有挑战性:自动递减解码器往往可以在不使用潜在空间的情况下解释数据,这种潜在空间被称为后台崩溃。为了减轻这一影响,最先进的模型通过对解码器输入采用统一随机抽出的方法,削弱了强大的解码器。我们从理论上表明,这删除了由解码器输入提供的、通过利用潜在空间加以补偿的有分辨的相互信息。我们随后提出了一个对抗性培训战略,以实现基于信息的随机性辍学。与标准文本基准数据集的统一辍学相比,我们的定向方法提高了在标准文本基准数据集中的排序性能和在潜在空间中获取的信息。