In this paper, the placement strategy design of coded caching in fog-radio access networks (F-RANs) is investigated. By considering time-variant content popularity, federated deep reinforcement learning is exploited to learn the placement strategy for our coded caching scheme. Initially, the placement problem is modeled as a Markov decision process (MDP) to capture the popularity variations and minimize the long-term content access delay. The reformulated sequential decision problem is solved by dueling double deep Q-learning (dueling DDQL). Then, federated learning is applied to learn the relatively low-dimensional local decision models and aggregate the global decision model, which alleviates over-consumption of bandwidth resources and avoids direct learning of a complex coded caching decision model with high-dimensional state space. Simulation results show that our proposed scheme outperforms the benchmarks in reducing the content access delay, keeping the performance stable, and trading off between the local caching gain and the global multicasting gain.

翻译：在本文中,对雾射存取网络(F-RANs)中编码缓存的定位战略设计进行了调查。在考虑时间差异内容受欢迎度的同时,利用联盟深度强化学习来学习我们编码缓存办法的定位战略。最初,安置问题模拟为Markov决策程序(MDP),以捕捉流行变异并尽量减少长期内容存取延迟。重订顺序决定问题通过双深度Q学习(裁断DDQL)来解决。然后,采用联合学习来学习相对低维的地方决定模式并汇总全球决定模式,以缓解带宽资源的过度消耗,避免直接学习带有高维状态空间的复杂编码缓存决定模式。模拟结果表明,我们提议的计划超过了减少内容存取延迟的基准,保持性能稳定,并在本地缓存收益和全球多投产收益之间进行交易。