Decentralized Federated Learning by Partial Message Exchange.

Abstract

Decentralized federated learning (DFL) has emerged as a transformative server-free paradigm that enables collaborative learning over large-scale heterogeneous networks. However, it continues to face fundamental challenges, including data heterogeneity, restrictive assumptions for theoretical analysis, and de graded convergence when standard communication- or privacy enhancing techniques are applied. To overcome these drawbacks, this paper develops a novel algorithm, PaME (DFL by Partial Message Exchange). The central principle is to allow only randomly selected sparse coordinates to be exchanged between two neighbor nodes. As a result, PaME significantly reduces communication costs while simultaneously limiting the exposure of data-sensitive information during transmission. The latter property is rigorously characterized by a formal reconstruction risk theory under partial observation. Moreover, the algorithm is proven to converge in expectation to a stationary point at a linear rate, provided that the gradient is locally Lipschitz continuous and the communication matrix is doubly stochastic. These two mild assumptions not only dispenses with many restrictive conditions commonly imposed by existing DFL methods but also enables PaME to effectively address data heterogeneity. Furthermore, comprehensive numerical experiments demonstrate its superior performance compared with several representative decentralized learning algorithms.