The accompanying actions and gestures in dialogue are often closely linked to interactions with the environment, such as looking toward the interlocutor or using gestures to point to the described target at appropriate moments. Speech and semantics guide the production of gestures by determining their timing (WHEN) and style (HOW), while the spatial locations of interactive objects dictate their directional execution (WHERE). Existing approaches either rely solely on descriptive language to generate motions or utilize audio to produce non-interactive gestures, thereby lacking the characterization of interactive timing and spatial intent. This significantly limits the applicability of conversational gesture generation, whether in robotics or in the fields of game and animation production. To address this gap, we present a full-stack solution. We first established a unique data collection method to simultaneously capture high-precision human motion and spatial intent. We then developed a generation model driven by audio, language, and spatial data, alongside dedicated metrics for evaluating interaction timing and spatial accuracy. Finally, we deployed the solution on a humanoid robot, enabling rich, context-aware physical interactions.

翻译：对话中的伴随动作和手势通常与环境交互紧密相关，例如在适当时刻看向对话者或使用手势指向所描述的目标。语音和语义通过确定手势的时机（何时）和风格（如何）来引导手势的生成，而交互对象的空间位置则决定了其方向性执行（何地）。现有方法要么仅依赖描述性语言生成动作，要么利用音频产生非交互式手势，从而缺乏对交互时机和空间意图的表征。这显著限制了对话手势生成的适用性，无论是在机器人学还是在游戏与动画制作领域。为弥补这一不足，我们提出了一套全栈解决方案。我们首先建立了一种独特的数据采集方法，以同时捕获高精度人体运动与空间意图。随后，我们开发了一个由音频、语言和空间数据驱动的生成模型，并设计了专门用于评估交互时机和空间准确性的度量指标。最后，我们将该解决方案部署于一个人形机器人上，实现了丰富且具有情境感知的物理交互。