Video prediction is a fundamental task for various downstream applications, including robotics and world modeling. Although general video prediction models have achieved remarkable performance in standard scenarios, occlusion is still an inherent challenge in video prediction. We hypothesize that providing explicit information about motion (via point-flow) and geometric structure (via depth-maps) will enable video prediction models to perform better in situations with occlusion and the background motion. To investigate this, we present the first systematic study dedicated to occluded video prediction. We use a standard multi-object latent transformer architecture to predict future frames, but modify this to incorporate information from depth and point-flow. We evaluate this model in a controlled setting on both synthetic and real-world datasets with not only appearance-based metrics but also Wasserstein distances on object masks, which can effectively measure the motion distribution of the prediction. We find that when the prediction model is assisted with point flow and depth, it performs better in occluded scenarios and predicts more accurate background motion compared to models without the help of these modalities.

翻译：视频预测是机器人学和世界建模等多种下游应用的基础任务。尽管通用视频预测模型在标准场景中已取得显著性能，但遮挡问题仍是视频预测中固有的挑战。我们假设，通过提供关于运动（通过点流）和几何结构（通过深度图）的显式信息，视频预测模型能够在存在遮挡和背景运动的情况下表现更优。为验证此假设，我们首次开展了针对遮挡视频预测的系统性研究。我们采用标准的多对象潜在Transformer架构来预测未来帧，但对其进行改进以融合深度与点流信息。我们在受控环境下，使用合成数据集和真实世界数据集对该模型进行评估，不仅采用基于外观的指标，还计算对象掩码上的Wasserstein距离，该距离能有效衡量预测结果的运动分布。研究发现，当预测模型获得点流和深度信息辅助时，其在遮挡场景中的表现优于未使用这些模态的模型，并能更准确地预测背景运动。