面向非独立同分布数据的自适应联邦深度学习算法

张泽辉; 李庆丹; 富瑶; 何宁昕; 高铁杠

doi:10.16383/j.aas.c201018

面向非独立同分布数据的自适应联邦深度学习算法

doi: 10.16383/j.aas.c201018

1.
南开大学软件学院天津 300071

基金项目: 国家科技重大专项(2018YFB0204304), 天津市研究生科研创新基金(2019YJSB067)资助

详细信息

作者简介:
张泽辉：南开大学软件学院博士研究生. 2019年获得武汉理工大学工学硕士学位. 主要研究方向为联邦学习,,故障诊断和智能船舶控制. E-mail: zhangtianxia918@163.com

李庆丹：南开大学软件学院硕士研究生. 主要研究方向为图像加密, 信息安全. E-mail: lqd18812745024@163.com

富瑶：南开大学软件学院硕士研究生. 主要研究方向为云端数据完整性验证, 信息安全. E-mail: FuYao_TJ@163.com

何宁昕：南开大学软件学院硕士研究生.2020年获得河北经贸大学工学学士学位. 主要研究方向为信息安全, 联邦学习. Email: ningxinhe1998@163.com

高铁杠：南开大学软件学院教授, 1991获华中理工大学应用数学专业硕士学位, 2005获南开大学理学博士学位. 主要研究方向为联邦学习, 信息隐藏和云端数据安全. 本文通信作者. E-mail: gaotiegang@nankai.edu.cn

计量
- 文章访问数: 2190
- HTML全文浏览量: 1457
- 被引次数: 0
出版历程
- 收稿日期: 2020-12-08
- 录用日期: 2021-03-19
- 网络出版日期: 2021-06-04

Adaptive Federated Deep Learning With Non-IID Data

1.
College of Software, Nankai University, Tianjin 300071

Funds: Supported by National Science and Technology Major Project of China (2018YFB0204304), Tianjin Research Innovation Project for Postgraduate Students (2019YJSB067)

More Information

Author Bio:
ZHANG Ze-Hui　Ph. D. candidate at the College of Software, Nankai University. He received his master degree from Wuhan University of Technology in 2019. His research interest covers federated learning, fault diagnosis and intelligent ship control

LI Qing-Dan　Master student at the College of Software, NanKai University. Her research interest covers image encryption and information security

FU Yao　Master student at the College of Software, NanKai University. Her research interest covers cloud data integrity verification and information security

HE Ning-Xin　Master student at the College of Software, Nankai University. She received her B.Sc degree from Hebei University of Economics and Business in 2020. Her research interest covers information security and federated learning

GAO Tie-Gang　Professor at the College of Software, Nankai University. He received his master degree in Applied Mathematics from Huazhong University of Science and Technology in 1991, and Ph.D. degree in Nankai University in 2005. His research interest covers federated learning, image watermarking, information hiding and cloud data security

摘要

摘要: 近些年, 联邦学习由于能够打破数据壁垒, 实现孤岛数据价值变现, 受到了工业界和学术界的广泛关注. 然而, 在实际工程应用中, 联邦学习存在着数据隐私泄露和模型性能损失的问题. 对此, 首先对这两个问题进行了数学描述与分析. 然后, 提出了一种自适应模型聚合方案, 该方案能够设定各参与者的mini-batch值和自适应调整全局模型聚合间隔, 旨在保证模型精度的同时, 提高联邦学习训练效率. 并且, 混沌系统被首次引入联邦学习领域中, 用于构建一种基于混沌系统和同态加密的混合隐私保护方案, 从而进一步提升系统的隐私保护水平. 理论分析与实验结果表明, 提出的联邦学习算法能够保证参与者的数据隐私安全. 并且, 在非独立同分布数据的场景下, 该算法够在保证模型精度的前提下提高训练效率, 降低系统通信成本, 具备实际工业场景应用的可行性.
- 联邦学习 /
- 深度学习 /
- 隐私保护 /
- 同态加密 /
- 混沌系统
Abstract: In recent years, federated learning that can break data barriers and realize the value of isolated data, has been received wide-spread attention from industry and academia. However, in real industry applications, federated learning has problems with privacy leakage and model accuracy loss, which is analyzed through mathematical demonstration in this study. To solve the issues, this paper proposes an adaptive global model aggregation scheme that can adaptively set the mini-batch value of each participant and the global model aggregation interval for the parameter server, which aims to improve the training efficiency while ensuring the accuracy of the model. Moreover, this paper introduces the chaos system into the federated learning field, which is used to construct a hybrid privacy protection scheme based on chaos system and homomorphic encryption, thereby further improving the privacy protection level. Theoretical analysis and experimental results show that the proposed approach can guarantee the data privacy security of participants. Moreover, in the non-independent and identically distributed (Non-IID) data scenario, the proposed method can improve the training efficiency and reduce communication cost while ensuring the model accuracy, which is feasible for real industrial applications.
- Federated learning /
- deep learning /
- privacy-preserving /
- homomorphic encryption /
- chaos system

HTML全文

图 1 多层神经网络模型

Fig. 1 Multi-layer neural network model

下载: 全尺寸图片幻灯片

图 2 不同比例数据泄露的图片

Fig. 2 Images of different proportion data leakage

下载: 全尺寸图片幻灯片

图 3 本文所提出的联邦学习系统结构图

Fig. 3 The structure diagram of the proposed federated learning system

下载: 全尺寸图片幻灯片

图 4 联邦学习训练过程交互图

Fig. 4 Interaction diagram of the federated learning system

下载: 全尺寸图片幻灯片

图 5 混沌加密后的参数推理实验图

Fig. 5 Inferring data of the encrypted parameters

下载: 全尺寸图片幻灯片

图 6 在CIFAR10上不同联邦学习模型的实验曲线

Fig. 6 Experiment curves of the different federated learning models on CIFAR10

下载: 全尺寸图片幻灯片

图 7 在F-MNIST上不同联邦学习模型的实验曲线

Fig. 7 Experiment curves of the different federated learning models on F-MNIST

下载: 全尺寸图片幻灯片

图 8 在CIFAR-10和F-MNIST数据集的mini-batch设定消融实验曲线

Fig. 8 Experiment curves of the minibatch size setting on CIFAR-10 and F-MNIST

下载: 全尺寸图片幻灯片

图 9 CIFAR-10自适应模型更新间隔消融实验曲线

Fig. 9 Experiment curves of the adaptive model aggregation interval on CIFAR-10

下载: 全尺寸图片幻灯片

图 10 F-MNIST自适应模型更新间隔消融实验曲线

Fig. 10 Experiment curves of the adaptive model aggregation interval on F-MNIST

下载: 全尺寸图片幻灯片

表 1 加密/解密算法的执行时间

Table 1 Execution time of the encryption/decryption operations

参数个数	500	2000	54000
随机数生成	12.05 ms	25.50 ms	0.40 s
CKKS加密	9.37 ms	9.68 ms	0.54 s
CKKS解密	1.56 ms	17.18 ms	0.03 s
CKKS密文加法	0.15 ms	0.15 ms	0.02 s
Paillier加密	3.82 s	14.61 s	410.32 s
Paillier解密	1.06 s	4.22 s	115.92 s
Paillier密文加法	7.87 ms	30.03 ms	0.87 s

下载: 导出CSV

表 2 加密/解密算法的执行次数

Table 2 Execution numbers of the encryption/decryption operations

模型更新间隔$\tau $	50	80	100
$\tau=1$^[18]	7800	12500	15600
$\tau=4$^[16]	1550	2480	3100
$\tau=15$^[33]	500	800	1000

下载: 导出CSV

表 3 不同联邦学习方案的功能对比

Table 3 The functionality analysis of the different FLs

功能	PFL	AFL	MFL	APFL
隐私保护	√	×	×	√
自适应调整$\tau $	×	√	×	√
mini-batch设定	×	×	×	√
动量项加速	×	×	√	√

下载: 导出CSV

表 4 CIFAR10上不同联邦学习模型的分类结果

Table 4 Classification results of the different federated learning models on CIFAR10

Method	Accuracy	Precision	Recall	Dev_avg	聚合次数
CL	63.36%	63.92%	63.29%	—	—
FL($\tau $=15)^[33]	25.76%	9.34%	25.87%	49.91%	250
FL($\tau $=4)^[16]	27.64%	50.14%	27.76%	45.04%	1100
FL($\tau $=1)^[18]	61.78%	62.76%	61.77%	1.91%	4400
APFL	63.66%	63.49%	63.64%	2.02%	2758

下载: 导出CSV

表 5 F-MNIST上不同联邦学习模型的分类结果

Table 5 Classification results of the different federated learning models on F-MNIST

Method	Accuracy	Precision	Recall	Dev_avg	聚合次数
CL	90.15%	90.07%	90.15%	—	—
FL($\tau $=15)^[33]	65.99%	62.18%	65.99%	31.43%	350
FL($\tau $=4)^[16]	72.77%	65.24%	72.77%	23.16%	1350
FL($\tau $=1)^[18]	89.10%	89.25%	89.10%	0.88%	5250
APFL	89.36%	89.30%	89.36%	0.87%	2339

下载: 导出CSV

表 6 CIFAR-10下的mini-batch设定消融实验结果

Table 6 Ablation experiment results of the mini-batch size setting on CIFAR-10

Method	Accuracy	Precision	Recall	Dev_avg
CL	63.36%	63.92%	63.29%	—
FL($\tau $=15)^[33]	25.76%	9.34%	25.87%	49.91%
FL($\tau $=15)-Mbs	25.70%	9.14%	25.78%	50.07%
FL($\tau $=4)	27.64%	50.14%	27.76%	45.04%
FL($\tau $=4)-Mbs	63.66%	60.93%	36.06%	32.90%
FL($\tau $=1)^[18]	61.78%	62.76%	61.77%	1.91%
FL($\tau $=1)-Mbs	63.02%	64.08%	62.27%	1.53%

下载: 导出CSV

表 7 F-MNIST下的mini-batch设定消融实验结果

Table 7 Ablation experiment results of the mini-batch size setting on F-MNIST

Method	Accuracy	Precision	Recall	Dev_avg
CL	90.15%	90.07%	90.15%	—
FL($\tau $=15)^[33]	65.99%	62.18%	65.99%	31.43%
FL($\tau $=15)-Mbs	69.99%	64.29%	69.99%	26.05%
FL($\tau $=4)	27.76%	50.14%	27.76%	45.04%
FL($\tau $=4)-Mbs	76.23%	84.84%	76.23%	14.85%
FL($\tau $=1)^[18]	89.10%	89.25%	89.10%	0.88%
FL($\tau $=1)-Mbs	89.27%	89.25%	89.27%	0.99%

下载: 导出CSV

表 8 CIFAR-10下的自适应更新间隔消融实验结果

Table 8 The ablation experiment results of the adaptive model aggregation interval on CIFAR-10

Method	Accuracy	Precision	Recall	Dev_avg	聚合次数
CL	63.36%	63.92%	63.29%	—	—
FL($\tau $ =15)^[33]	25.76%	9.34%	25.87%	49.91%	250
FL($\tau $=4)^[16]	27.64%	50.14%	27.76%	45.04%	1100
FL($\tau $=1)^[18]	61.78%	62.76%	61.77%	1.91%	4000
APFL(no mbs)	61.10%	62.00%	61.36%	3.27%	1742

下载: 导出CSV

表 9 F-MNIST下的自适应更新间隔消融实验结果

Table 9 Ablation experiment results of the adaptive model aggregation interval on F-MNIST

Method	Accuracy	Precision	Recall	Dev_avg	聚合次数
CL	90.15%	90.07%	90.15%	—	—
FL($\tau $=15)^[33]	65.99%	62.18%	65.99%	31.43%	250
FL($\tau $=4)^[16]	72.77%	65.24%	72.77%	23.16%	1100
FL($\tau $=1)^[18]	89.10%	89.25%	89.10%	0.88%	4400
APFL(no mbs)	89.48%	89.42%	89.48%	0.84%	1336

下载: 导出CSV

参考文献(42)

[1]	孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题. 自动化学报, 2020, 46(07): 1301-1312. Sun Chang-Yin, Mu Chao-Xu. Important scientific problems of multi-agent Deep Reinforcement Learning. Acta Automatica Sinica, 2020, 46(07): 1301-1312.
[2]	金侠挺, 王耀南, 张辉, 等. 基于贝叶斯CNN和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312-2327. Jin Xia-Ting, Wang Yao-Nan, Zhang Hui, et al. DeepRail: automatic visual detection system for railway surface defect using bayesian CNN and attention Network. Acta Automatica Sinica, 2019, 45(12): 2312-2327.
[3]	Zhang Z, Guan C, Liu Z. Real-time optimization energy management strategy for fuel cell hybrid ships considering power sources degradation[J]. IEEE Access, 2020, 8: 87046-87059. doi: 10.1109/ACCESS.2020.2991519
[4]	Chen H, Zhang Z, Guan C, et al. Optimization of sizing and frequency control in battery/supercapacitor hybrid energy storage system for fuel cell ship. Energy, 2020, 197: 117285. doi: 10.1016/j.energy.2020.117285
[5]	鲜征征, 李启良, 黄晓宇, 等. 基于差分隐私和SVD++的协同过滤算法. 控制与决策, 2019, 34(01): 43-54. Xian Zheng-Zheng, Li Qi-Liang, Huang Xiao-Yu, et al. Collaborative filtering via SVD++ with differential privacy. Control and Decision, 2019, 34(01): 43-54.
[6]	Jing L, Xk C, Sl A, et al. Privacy preservation for machine learning training and classification based on homomorphic encryption schemes. Information Sciences, 2020, 526: 166-179. doi: 10.1016/j.ins.2020.03.041
[7]	Gong M, Pan K, Xie Y, et al. Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition. Neural Networks, 2020, 125: 131-141. doi: 10.1016/j.neunet.2020.02.001
[8]	张超, 李强, 陈子豪, 等. Medical Chain: 联盟式医疗区块链系统. 自动化学报, 2019, 45(08): 1495-1510. Zhang Chao, Li Qiang, Chen Zi-Hao, et al. Medical Chain: alliance medical blockchain system. Acta Automatica Sinica, 2019, 45(08): 1495-1510.
[9]	Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
[10]	Li T, Sahu A K, Talwalkar A, et al. Federated learn-ing: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50-60. doi: 10.1109/MSP.2020.2975749
[11]	Zhang W, Li X, Ma H, et al. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. Knowledge-Based Systems, 2021, 213: 106679. doi: 10.1016/j.knosys.2020.106679
[12]	Sheller M J, Edwards B, Reina G A, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 2020, 10(1): 1-12. doi: 10.1038/s41598-019-56847-4
[13]	Kwon D, Jeon J, Park S, et al. Multiagent DDPG-Based Deep Learning for Smart Ocean Federated Learning IoT Networks. IEEE Internet of Things Journal, 2020, 7(10): 9895-9903. doi: 10.1109/JIOT.2020.2988033
[14]	Rothchild D, Panda A, Ullah E, et al. Fetchsgd: Communication-efficient federated learning with sketching[C]//International Conference on Machine Learning. PMLR, 2020: 8253−8265.
[15]	Duan M, Liu D, Chen X, et al. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 2020, 32(1): 59-71.
[16]	Liu W, Chen L, Chen Y, et al. Accelerating Federated Learning via Momentum Gradient Descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754-1766. doi: 10.1109/TPDS.2020.2975189
[17]	Wang S, Tuor T, Salonidis T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205-1221. doi: 10.1109/JSAC.2019.2904348
[18]	Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345.
[19]	张泽辉, 富瑶, 高铁杠. 支持数据隐私保护的联邦深度神经网络模型研究. 自动化学报, 2020: 1-14. Zhang Ze-Hui, Fu Yao, Gao Tie-Gang. Research on federated deep neural network model for data privacy protection. Acta Automatica Sinica, 2020: 1-14.
[20]	Lyu L, Li Y, Nandakumar K, et al. How to democratise and protect AI: fair and differentially private decentralised deep learning. IEEE Transactions on Dependable and Secure Computing, 2020.
[21]	Wang Y, Gu M, Ma J, et al. DNN-DP: Differential Privacy Enabled Deep Neural Network Learning Framework for Sensitive Crowdsourcing Data. IEEE Transactions on Computational Social Systems, 2019, 7(1): 215-224.
[22]	Carpov S, Gama N, Georgieva M, et al. Privacy-preserving semi-parallel logistic regression training with Fully Homomorphic Encryption. 2019: 101.
[23]	宋蕾, 马春光, 段广晗, 等. 基于数据纵向分布的隐私保护逻辑回归. 计算机研究与发展, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414 Song Lei, Ma Chun-Guang, Duan Guang-Han, et al. Privacy-preserving logistic regression on vertically partitioned data. Computer Research and Development, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414
[24]	Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345.
[25]	Ou W, Zeng J, Guo Z, et al. A homomorphic-encryption-based vertical federated learning scheme for rick management. Computer Science and Information Systems, 2020, 17(3): 819-834. doi: 10.2298/CSIS190923022O
[26]	Chen H, Chillotti I, Song Y. Improved Bootstrapping for Approximate Homomorphic Encryption. Springer, Cham, 2019.
[27]	Xiao X, Wu T, Chen Y, et al. Privacy-Preserved Ap-proximate Classification Based on Homomorphic En-cryption. Mathematical and Computational Applications, 2019, 24(4): 92. doi: 10.3390/mca24040092
[28]	Zehui Z, Fu Y, Gao T. A Hybrid Image Encryption Algorithm Based on Chaos System and Simplified Ad-vanced Encryption System. International Journal of Multimedia Data Engineering and Management (IJMDEM), 2020, 11(4): 1-24. doi: 10.4018/IJMDEM.2020100101
[29]	Luo Y, Yu J, Lai W, et al. A novel chaotic image encryption algorithm based on improved baker map and logistic map. Multimedia Tools and Applications, 2019, 78(15): 22023-22043. doi: 10.1007/s11042-019-7453-3
[30]	Sathiyamurthi P, Ramakrishnan S. Speech encryption algorithm using FFT and 3D-Lorenz–logistic chaotic map. Multimedia Tools and Applications, 2020, 79(3).
[31]	Sattler F, Müller K, Samek W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
[32]	Al-Sharman M, Murdoch D, Cao D, et al. A sensor-less state estimation for a safety-oriented cyber-physical system in urban driving: deep learning approach. IEEE/CAA Journal of Automatica Sinica, 2020.
[33]	Weng J, Weng J, Zhang J, et al. Deepchain: Auditable and privacy-preserving deep learning with blockchain-based incentive. IEEE Transactions on Dependable and Secure Computing, 2019.
[34]	Sattler F, Wiedemann S, Müller K, et al. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
[35]	Xu G, Li H, Zhang Y, et al. Privacy-preserving federated deep learning with irregular users. IEEE Transactions on Dependable and Secure Computing, 2020.
[36]	Teng S, Wu N, Zhu H, et al. SVM-DT-based adaptive and collaborative intrusion detection. IEEE/CAA Journal of Automatica Sinica, 2017, 5(1): 108-118.
[37]	Wang F Y, Fundamental Issues in Research of Computing with Words and Linguistic Dynamic Systems. Acta Automatica Sinica (Periodical style), 2005, 31(6): 844--852.
[38]	Roychoudhury R, Bandyopadhyay S, Paul K. Adistributed mechanism for topology discovery in ad hoc wireless networks using mobile agents. In: Proceeding of IEEE First Annual Workshop on Mobile and Ad hoc Networking and Computing (Conference Proceedings style), Piscataway, USA: IEEE Press, 2000.145—146.
[39]	Hryniewicz O. An evaluation of the reliability of complex systems using shadowed sets and fuzzy lifetime data. International Journal of Automation and Computing (Periodical style—Accepted for publication), to be published.
[40]	Zhang W. Reinforcement Learning for Job-Shop Scheduling. [Ph. D. Dissertation], Oregon State University, 1996.
[41]	IEEE Criteria for Class IE Electric Systems (Standards style), IEEE Standard 308, 1969.
[42]	Jones J. Networks, 2nd ed. (Online Sources style). [Online], available: http://www.atm.com, May 10, 1991.