We present Lang2Motion, a framework for language-guided point trajectory generation by aligning motion manifolds with joint embedding spaces. Unlike prior work focusing on human motion or video synthesis, we generate explicit trajectories for arbitrary objects using motion extracted from real-world videos via point tracking. Our transformer-based auto-encoder learns trajectory representations through dual supervision: textual motion descriptions and rendered trajectory visualizations, both mapped through CLIP's frozen encoders. Lang2Motion achieves 34.2% Recall@1 on text-to-trajectory retrieval, outperforming video-based methods by 12.5 points, and improves motion accuracy by 33-52% (12.4 ADE vs 18.3-25.3) compared to video generation baselines. We demonstrate 88.3% Top-1 accuracy on human action recognition despite training only on diverse object motions, showing effective transfer across motion domains. Lang2Motion supports style transfer, semantic interpolation, and latent-space editing through CLIP-aligned trajectory representations.

翻译：本文提出Lang2Motion框架，通过将运动流形与联合嵌入空间对齐，实现语言引导的点轨迹生成。与以往专注于人体运动或视频合成的研究不同，我们利用通过点跟踪从真实世界视频中提取的运动，为任意物体生成显式轨迹。基于Transformer的自编码器通过双重监督学习轨迹表示：文本运动描述和渲染的轨迹可视化，两者均通过CLIP的冻结编码器进行映射。Lang2Motion在文本到轨迹检索任务中实现了34.2%的Recall@1，比基于视频的方法高出12.5个百分点，并且与视频生成基线相比，运动精度提高了33-52%（ADE为12.4，基线为18.3-25.3）。尽管仅在多样化物体运动数据上训练，我们在人类动作识别任务中达到了88.3%的Top-1准确率，显示出跨运动领域的有效迁移能力。Lang2Motion通过CLIP对齐的轨迹表示支持风格迁移、语义插值和潜在空间编辑。