BRFedTD: A Novel Framework for Federated Reinforcement Learning With Byzantine-Resilient Policy Evaluation.

Abstract

Federated reinforcement learning (FRL) enables distributed agents to collaboratively evaluate policies without sharing raw data, making it a promising approach for privacy-preserving and scalable decision-making. However, the presence of Byzantine agents, which may behave arbitrarily or maliciously, poses significant challenges to the robustness and reliability of FRL. To address this issue, in this paper, we propose a novel trimmed mean-based robust federated policy evaluation framework called Byzantine-resilient federated temporal difference learning (BRFedTD), and establish a finite-time convergence theory of BRFedTD. This framework effectively addresses the combined challenges of linear function approximation, heterogeneous Markov decision processes (MDPs), multiple local updates, and robust aggregation. To support more accurate confidence interval estimation and policy uncertainty analysis, we further derive the asymptotic distribution of the estimation error, showing that BRFedTD achieves asymptotic normality and efficiency in the case of identical MDPs without Byzantine attacks. This represents, to the best of our knowledge, the first asymptotic normality result established in FRL. Extensive numerical experiments demonstrate the robustness and effectiveness of the proposed algorithm, and corroborate that it generalizes to deep reinforcement learning and performs well on complex control tasks.