Privacy-preserving machine learning has become a key conundrum for multi-party artificial intelligence. Federated learning (FL) and Split Learning (SL) are two frameworks that enable collaborative learning while keeping the data private (on device). In FL, each data holder trains a model locally and releases it to a central server for aggregation. In SL, the clients must release individual cut-layer activations (smashed data) to the server and wait for its response (during both inference and back propagation). While relevant in several settings, both of these schemes have a high communication cost, rely on server-level computation algorithms and do not allow for tunable levels of collaboration. In this work, we present a novel approach for privacy-preserving machine learning, where the clients collaborate via online knowledge distillation using a contrastive loss (contrastive w.r.t. the labels). The goal is to ensure that the participants learn similar features on similar classes without sharing their input data. To do so, each client releases averaged last hidden layer activations of similar labels to a central server that only acts as a relay (i.e., is not involved in the training or aggregation of the models). Then, the clients download these last layer activations (feature representations) of the ensemble of users and distill their knowledge in their personal model using a contrastive objective. For cross-device applications (i.e., small local datasets and limited computational capacity), this approach increases the utility of the models compared to independent learning and other federated knowledge distillation (FD) schemes, is communication efficient and is scalable with the number of clients. We prove theoretically that our framework is well-posed, and we benchmark its performance against standard FD and FL on various datasets using different model architectures.

翻译：保存隐私的机器学习已成为多党人工智能的关键难题。 联邦学习( FL) 和 Split Learning (SL) 是两个框架, 使得在保持数据私密( 设备上) 的同时能够合作学习。 在 FL 中, 每个数据持有者都在当地培训模型, 并将模型发布给服务器, 并等待其响应( 在推断和回传中) 。 虽然在几个环境中, 联邦学习( Fl) 和 Split Learning (SL) 是两个框架, 使得能够合作学习, 同时又能保持数据私密( 设备上) 。 在 FL 中, 每个数据持有者通过在线知识蒸馏将模型( 动态 w.r. t. 标签) 向服务器发布单个剪贴( 缩放数据) 。 目标是确保参与者在类似课程中学习相似的功能( 在不共享输入模型数据的数据数据数据数据数据流数据流中, 每个客户平均和跨层启动类似标签, 而中央服务器仅作为转发器( irealalal dial dial dial dialation) distration distration distration) distration (istration) listration) list list list list