Experience replay (ER) improves the data efficiency of off-policy reinforcement learning (RL) algorithms by allowing an agent to store and reuse its past experiences in a replay buffer. While many techniques have been proposed to enhance ER by biasing how experiences are sampled from the buffer, thus far they have not considered strategies for refreshing experiences inside the buffer. In this work, we introduce Lucid Dreaming for Experience Replay (LiDER), a conceptually new framework that allows replay experiences to be refreshed by leveraging the agent's current policy. LiDER consists of three steps: First, LiDER moves an agent back to a past state. Second, from that state, LiDER then lets the agent execute a sequence of actions by following its current policy---as if the agent were "dreaming" about the past and can try out different behaviors to encounter new experiences in the dream. Third, LiDER stores and reuses the new experience if it turned out better than what the agent previously experienced, i.e., to refresh its memories. LiDER is designed to be easily incorporated into off-policy, multi-worker RL algorithms that use ER; we present in this work a case study of applying LiDER to an actor-critic based algorithm. Results show LiDER consistently improves performance over the baseline in six Atari 2600 games. Our open-source implementation of LiDER and the data used to generate all plots in this work are available at github.com/duyunshu/lucid-dreaming-for-exp-replay.

翻译：经验重现( ER) 提高了政策外强化学习( RL) 算法的数据效率 。 LIDER 包括三个步骤 : 首先, liDER 将一个代理商转回到过去状态 。 其次, 利DER 然后让该代理商执行一系列行动, 遵循它目前的政策-, 因为如果该代理商对过去有“ 梦想” 的感觉, 并且可以尝试不同的行为来体验这个梦中的新经验 。 第三, 利DER 商店和再利用新的经验, 如果它比代理商以往所经历的更好的话。 利DER 包括三个步骤 : 首先, 利DER 将一个代理商转回到过去状态 。 其次, 利DER 然后让该代理商执行一系列行动, 遵循它目前的政策-, 经验从缓冲到他们没有考虑过在缓冲中更新经验 。 第三, 利DER 存储和再利用新的经验, 如果它比代理商以往所经历的更好。 即, 更新它的记忆。 。 利DER 设计可以很容易在本次基于Li- IMDER 的运行的运行中, 的运行运行中, 在本次的运行中, 应用一个基于我们数据库的运行中, 的运行中,, 不断的运行中, 改进了一个基于我们 的运行中, 的运行局 的运行中, 。