One of the key challenges to deep reinforcement learning (deep RL) is to ensure safety at both training and testing phases. In this work, we propose a novel technique of unsupervised action planning to improve the safety of on-policy reinforcement learning algorithms, such as trust region policy optimization (TRPO) or proximal policy optimization (PPO). We design our safety-aware reinforcement learning by storing all the history of "recovery" actions that rescue the agent from dangerous situations into a separate "safety" buffer and finding the best recovery action when the agent encounters similar states. Because this functionality requires the algorithm to query similar states, we implement the proposed safety mechanism using an unsupervised learning algorithm, k-means clustering. We evaluate the proposed algorithm on six robotic control tasks that cover navigation and manipulation. Our results show that the proposed safety RL algorithm can achieve higher rewards compared with multiple baselines in both discrete and continuous control problems. The supplemental video can be found at: https://youtu.be/AFTeWSohILo.

翻译：深入强化学习( 深层强化学习) 的关键挑战之一是确保培训和测试阶段的安全。 在这项工作中,我们提议了一种不受监督的行动规划新方法,以提高政策强化学习算法的安全性,如信任区域政策优化(TRPO)或准政策优化(PPO)等。我们设计了安全意识强化学习方法,将从危险情形中拯救该物剂的所有“恢复”行动历史储存在单独的“安全”缓冲中,并在该物剂遇到类似状态时找到最佳恢复行动。由于这一功能需要算法来查询类似的州,我们使用一种不受监督的学习算法(k- means群)来执行拟议的安全机制。我们评估了涵盖导航和操纵的六种机器人控制任务的拟议算法。我们的结果显示,与离散和连续控制问题的多个基线相比,拟议的安全RL算法可以获得更高的回报。 补充视频见: https://youtu.be/AFTeWESohiLolo。