We investigate the discounting mismatch in actor-critic algorithm implementations from a representation learning perspective. Theoretically, actor-critic algorithms usually have discounting for both actor and critic, i.e., there is a $\gamma^t$ term in the actor update for the transition observed at time $t$ in a trajectory and the critic is a discounted value function. Practitioners, however, usually ignore the discounting ($\gamma^t$) for the actor while using a discounted critic. We investigate this mismatch in two scenarios. In the first scenario, we consider optimizing an undiscounted objective $(\gamma = 1)$ where $\gamma^t$ disappears naturally $(1^t = 1)$. We then propose to interpret the discounting in critic in terms of a bias-variance-representation trade-off and provide supporting empirical results. In the second scenario, we consider optimizing a discounted objective ($\gamma < 1$) and propose to interpret the omission of the discounting in the actor update from an auxiliary task perspective and provide supporting empirical results.

翻译：我们从代表性学习的角度来调查行为体-批评算法实施中的贴现错配。理论上,行为体-批评算法通常对演员和评论家都有折扣,也就是说,当时间观察到的转型时,在演员更新中有一个美元(gamma美元)的术语,在轨迹中,批评家是一个折扣价值功能。然而,从业者通常忽略对演员的折扣(gamma美元),而使用折扣评论家。我们用两种假设来调查这一错配。在第一种假设中,我们考虑优化一个未贴现的目标$(\gamma = 1美元),即$(gamma = 1美元)自然消失。我们然后提议用偏差-差异-代表交易来解释批评折扣,并提供经验结果支持。在第二种假设中,我们考虑优化一个贴现目标($\gamma < 1美元),并提议从辅助任务角度来解释在演员更新中忽略贴现,并提供经验结果支持。