Masked Autoencoders (MAEs) learn generalizable representations for image, text, audio, video, etc., by reconstructing masked input data from tokens of the visible data. Current MAE approaches for videos rely on random patch, tube, or frame-based masking strategies to select these tokens. This paper proposes AdaMAE, an adaptive masking strategy for MAEs that is end-to-end trainable. Our adaptive masking strategy samples visible tokens based on the semantic context using an auxiliary sampling network. This network estimates a categorical distribution over spacetime-patch tokens. The tokens that increase the expected reconstruction error are rewarded and selected as visible tokens, motivated by the policy gradient algorithm in reinforcement learning. We show that AdaMAE samples more tokens from the high spatiotemporal information regions, thereby allowing us to mask 95% of tokens, resulting in lower memory requirements and faster pre-training. We conduct ablation studies on the Something-Something v2 (SSv2) dataset to demonstrate the efficacy of our adaptive sampling approach and report state-of-the-art results of 70.0% and 81.7% in top-1 accuracy on SSv2 and Kinetics-400 action classification datasets with a ViT-Base backbone and 800 pre-training epochs.

翻译：蒙面自动显示器( MAEs) 学习图像、 文本、 音频、 视频等的可概括化表达式。 通过重建可见数据的标记, 从可见数据的标记中重建隐藏的输入数据。 目前的 MAE 视频方法依靠随机的补丁、 管式或基于框架的遮罩策略来选择这些标记。 本文建议 AdaMAE, 这是一种适合MAE 的代用遮罩策略, 是一种端到端的训练。 我们的适应性遮罩战略样本, 以语义背景为基础, 使用辅助取样网络 。 这个网络估计了空间时间批量符号的绝对分布。 增加预期重建错误的标记得到奖励, 并被选为可见的标记。 我们显示, AdaMAE 样本中有更多的来自高波面信息区域的标记, 从而可以遮盖95%的代用品, 从而降低记忆要求, 并加快培训前。 我们对 Somen v2 ( SSv2) 数据集, 以显示我们适应性取样方法的功效, 以及报告 SS- -10.0 和 KIM2 之前 的 State- brealestalationalational 数据 IP 7: 80.0 和 Kinalest- bregrealation IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP IP li% 1 810 和 mal IP IP IP IP IP IP IP IP IP IP IP IP IP 18%