We present KeyCLD, a framework to learn Lagrangian dynamics from images. Learned keypoints represent semantic landmarks in images and can directly represent state dynamics. Interpreting this state as Cartesian coordinates coupled with explicit holonomic constraints, allows expressing the dynamics with a constrained Lagrangian. Our method explicitly models kinetic and potential energy, thus allowing energy based control. We are the first to demonstrate learning of Lagrangian dynamics from images on the dm_control pendulum, cartpole and acrobot environments. This is a step forward towards learning Lagrangian dynamics from real-world images, since previous work in literature was only applied to minimalistic images with monochromatic shapes on empty backgrounds. Please refer to our project page for code and additional results: https://rdaems.github.io/keycld/

翻译：我们展示了 KeyCLD, 这是从图像中学习 Lagrangian 动态的框架 。 学习的关键点代表图像中的语义标志, 可以直接代表国家动态 。 将这个状态解读为Cartesian 坐标, 并配以明确的 Holonomic 限制, 允许用受限制的 Lagrangian 来表达动态 。 我们的方法明确模拟动能和潜在能量, 从而允许基于能源的控制 。 我们首先展示了从 dm_ controcure pentum、 wolpole 和 acrobot 环境中的图像中学习 Lagrangian 动态。 这是向从真实世界图像中学习 Lagrangian 动态迈出的一步, 因为先前的文学工作只应用在空背景中带有单色形状的最小化图像 。 请参见我们的项目页面的代码和额外结果 : https://rdaems. github. io/ keycld/ 。