Recent advancements in foundation models for 2D vision have substantially improved the analysis of dynamic scenes from monocular videos. However, despite their strong generalization capabilities, these models often lack 3D consistency, a fundamental requirement for understanding scene geometry and motion, thereby causing severe spatial misalignment and temporal flickering in complex 3D environments. In this paper, we present Motion4D, a novel framework that addresses these challenges by integrating 2D priors from foundation models into a unified 4D Gaussian Splatting representation. Our method features a two-part iterative optimization framework: 1) Sequential optimization, which updates motion and semantic fields in consecutive stages to maintain local consistency, and 2) Global optimization, which jointly refines all attributes for long-term coherence. To enhance motion accuracy, we introduce a 3D confidence map that dynamically adjusts the motion priors, and an adaptive resampling process that inserts new Gaussians into under-represented regions based on per-pixel RGB and semantic errors. Furthermore, we enhance semantic coherence through an iterative refinement process that resolves semantic inconsistencies by alternately optimizing the semantic fields and updating prompts of SAM2. Extensive evaluations demonstrate that our Motion4D significantly outperforms both 2D foundation models and existing 3D-based approaches across diverse scene understanding tasks, including point-based tracking, video object segmentation, and novel view synthesis. Our code is available at https://hrzhou2.github.io/motion4d-web/.

翻译：近期二维视觉基础模型的发展显著提升了对单目视频动态场景的分析能力。然而，尽管这些模型具有较强的泛化能力，却往往缺乏三维一致性——这是理解场景几何与运动的基本要求，导致在复杂三维环境中出现严重的空间错位与时间闪烁问题。本文提出Motion4D，一种通过将基础模型的二维先验知识整合到统一的四维高斯溅射表示中来应对这些挑战的新型框架。我们的方法采用两部分迭代优化框架：1）序列优化，通过连续阶段更新运动场与语义场以保持局部一致性；2）全局优化，联合优化所有属性以实现长期连贯性。为提升运动精度，我们引入了动态调整运动先验的三维置信度图，以及基于逐像素RGB误差与语义误差向欠表示区域插入新高斯点的自适应重采样过程。此外，通过交替优化语义场与更新SAM2提示的迭代细化过程，我们有效解决了语义不一致问题，从而增强了语义连贯性。大量实验评估表明，我们的Motion4D在点云跟踪、视频对象分割和新视角合成等多种场景理解任务中，均显著优于二维基础模型及现有三维方法。代码发布于https://hrzhou2.github.io/motion4d-web/。