In Federated Learning, a global model is learned by aggregating model updates computed at a set of independent client nodes, to reduce communication costs multiple gradient steps are performed at each node prior to aggregation. A key challenge in this setting is data heterogeneity across clients resulting in differing local objectives which can lead clients to overly minimize their own local objective, diverging from the global solution. We demonstrate that individual client models experience a catastrophic forgetting with respect to data from other clients and propose an efficient approach that modifies the cross-entropy objective on a per-client basis by re-weighting the softmax logits prior to computing the loss. This approach shields classes outside a client's label set from abrupt representation change and we empirically demonstrate it can alleviate client forgetting and provide consistent improvements to standard federated learning algorithms. Our method is particularly beneficial under the most challenging federated learning settings where data heterogeneity is high and client participation in each round is low.

翻译：在联邦学习中，通过汇总在一组独立客户节点上计算的模型更新来学习全局模型，以减少通信成本，每个节点在汇总之前执行多个梯度步长。此设置的一个关键挑战是客户端之间的数据异质性，导致本地目标不同，这可能导致客户端过度最小化其自己的本地目标，使其与全局解分散。我们证明单个客户端模型相对于来自其他客户端的数据存在灾难性忘记，并提出一种有效的方法，通过重新加权softmax对数，对每个客户端的交叉熵目标进行修改。该方法保护客户端标签集之外的类别免受突然的表示更改影响，并且我们通过实验证明它可以缓解客户端遗忘并为标准联合学习算法提供一致的改进。我们的方法在数据异质性高且每轮客户参与度低的最具挑战性的联邦学习设置下具有特殊益处。