Forecasting how human hands move in egocentric views is critical for applications like augmented reality and human-robot policy transfer. Recently, several hand trajectory prediction (HTP) methods have been developed to generate future possible hand waypoints, which still suffer from insufficient prediction targets, inherent modality gaps, entangled hand-head motion, and limited validation in downstream tasks. To address these limitations, we present a universal hand motion forecasting framework considering multi-modal input, multi-dimensional and multi-target prediction patterns, and multi-task affordances for downstream applications. We harmonize multiple modalities by vision-language fusion, global context incorporation, and task-aware text embedding injection, to forecast hand waypoints in both 2D and 3D spaces. A novel dual-branch diffusion is proposed to concurrently predict human head and hand movements, capturing their motion synergy in egocentric vision. By introducing target indicators, the prediction model can forecast the specific joint waypoints of the wrist or the fingers, besides the widely studied hand center points. In addition, we enable Uni-Hand to additionally predict hand-object interaction states (contact/separation) to facilitate downstream tasks better. As the first work to incorporate downstream task evaluation in the literature, we build novel benchmarks to assess the real-world applicability of hand motion forecasting algorithms. The experimental results on multiple publicly available datasets and our newly proposed benchmarks demonstrate that Uni-Hand achieves the state-of-the-art performance in multi-dimensional and multi-target hand motion forecasting. Extensive validation in multiple downstream tasks also presents its impressive human-robot policy transfer to enable robotic manipulation, and effective feature enhancement for action anticipation/recognition.

翻译：预测人类手部在第一人称视角下的运动对于增强现实和人机策略迁移等应用至关重要。近年来，已开发出多种手部轨迹预测（HTP）方法来生成未来可能的手部路径点，但这些方法仍存在预测目标不足、固有的模态鸿沟、手-头运动纠缠以及在下游任务中验证有限等问题。为应对这些局限性，我们提出了一个通用的手部运动预测框架，该框架考虑了多模态输入、多维度和多目标预测模式，以及面向下游应用的多任务功能。我们通过视觉-语言融合、全局上下文整合和任务感知的文本嵌入注入来协调多种模态，以预测手部在2D和3D空间中的路径点。提出了一种新颖的双分支扩散模型，用于同时预测人类头部和手部的运动，捕捉它们在第一人称视角下的运动协同性。通过引入目标指示器，预测模型除了广泛研究的手部中心点外，还能预测手腕或手指等特定关节的路径点。此外，我们使Uni-Hand能够额外预测手-物交互状态（接触/分离），以更好地促进下游任务。作为该领域首个纳入下游任务评估的工作，我们构建了新的基准来评估手部运动预测算法在现实世界中的适用性。在多个公开可用数据集及我们新提出的基准上的实验结果表明，Uni-Hand在多维度和多目标手部运动预测方面达到了最先进的性能。在多个下游任务中的广泛验证也展示了其出色的人机策略迁移能力，能够实现机器人操作，以及对动作预期/识别的有效特征增强。