Learning to generate diverse scene-aware and goal-oriented human motions in 3D scenes remains challenging due to the mediocre characteristics of the existing datasets on Human-Scene Interaction (HSI); they only have limited scale/quality and lack semantics. To fill in the gap, we propose a large-scale and semantic-rich synthetic HSI dataset, denoted as HUMANISE, by aligning the captured human motion sequences with various 3D indoor scenes. We automatically annotate the aligned motions with language descriptions that depict the action and the unique interacting objects in the scene; e.g., sit on the armchair near the desk. HUMANISE thus enables a new generation task, language-conditioned human motion generation in 3D scenes. The proposed task is challenging as it requires joint modeling of the 3D scene, human motion, and natural language. To tackle this task, we present a novel scene-and-language conditioned generative model that can produce 3D human motions of the desirable action interacting with the specified objects. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.

翻译：在3D场景中学习产生多样的景象感知和面向目标的人类运动仍然具有挑战性,因为现有的人类-系统互动(HISI)数据集具有平庸的特点;它们的规模/质量有限,缺乏语义。为了填补这一空白,我们建议采用一个大规模和语义丰富的合成HSI合成数据集,称为HIVISE,将所捕捉到的人类运动序列与各种3D室内场景相匹配。我们自动注意到与描述动作和独特互动对象的语言描述相一致的运动;例如,坐在办公桌旁的扶轮椅上。HIVISE因此使得新一代的任务得以在3D场景中产生以语言为条件的人类运动。拟议的任务具有挑战性,因为它需要联合建模3D场景、人类运动和自然语言。为了应对这项任务,我们提出了一个新型的场景和语言条件的基因化模型,可以产生3D人运动的3D运动,与指定对象相互作用。我们的实验表明,我们的模型在3D场景场景中产生多样化和具有逻辑一致性的人类运动。