Efficient Federated Learning over Heterogeneous Machines




Zhao, Lei

Journal Title

Journal ISSN

Volume Title



Decentralized data scattered among large number of different machines, therefore, there are two major challenges in large-scale machine learning, i.e., vast amount of data transmission over the communication network causing huge communication cost, and security and privacy concerns of the individual data. Federated Learning (FL) is a promising solution by enabling high-quality model training while keeping the data locally. We are dedicated to designing efficient FL schemes that benefit all participants. When engaging in collaborative learning with other machines, each machine must prioritize the interests of its owner. How to effectively collaborate among heterogeneous machines with diverse requirements to achieve powerful training results while maximizing the profits of each participant poses an interesting challenge in federated learning. The global model in FL attempts to acquire knowledge from each machine, while aggregating all local models into one optimal solution for every machine proves to be extremely challenging due to the heterogeneous demands. From an optimization standpoint, it becomes evident that each machine possesses its own objective function. As a result, the challenge arises as how to leverage the knowledge of others while obtaining the best possible solution for each machine. Our objective is to improve the efficiency of the collaboration of heterogeneous. The main challenges including the slow convergence speed and heavy communication cost which are caused by the heterogeneous local data sets, heterogeneous local feature spaces, and dynamic client participation. Our contributions are as follows. Firstly, the proposed transform-domain FL schemes based on Discrete Cosine Transform (DCT-FA) and Discrete Wavelet Transform (DWT-FA) aim to improve training efficiency and reduce communication burden for various IoT intelligence applications. The combination of frequency-domain and time-domain features approach (CDCT-FA) achieves higher test accuracy by combining time-domain and frequency-domain features. Secondly, the proposed collaborative learning framework for healthcare IoT devices with heterogeneous local feature spaces allows for privacy preservation while achieving performance similar to centralized training. The proposed central accelerated Federated Stochastic Variance Reduced Gradient (FSVRG) approach which in conjunction with a stochastic or deterministic client selection mechanism is shown to yield improved computational and communication efficiency contributing to the advancement of FL techniques by improving convergence speed with higher model accuracy. Additionally, the proposed data trading market and the integration of histogram of oriented gradients (HoG) with DWT approach address the challenges of data trading and non-stationarity of revenue patterns associated with data products.



Federated Learning, Central Acceleration, Stochastic and Deterministic Client Selection, Edge Intelligence, Conjugate Acceleration, Inexact Line Search, Data Trading, Transform-Domain, Autonomous Economic Agents, Heterogeneous Healthcare Informatics, Latent Features, Collaborative Learning, IoT intelligence applications