2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

面向非独立同分布数据的自适应联邦深度学习算法

张泽辉 李庆丹 富瑶 何宁昕 高铁杠

张泽辉, 李庆丹, 富瑶, 何宁昕, 高铁杠. 面向非独立同分布数据的自适应联邦深度学习算法. 自动化学报, 2021, x(x): 1−13 doi: 10.16383/j.aas.c201018
引用本文: 张泽辉, 李庆丹, 富瑶, 何宁昕, 高铁杠. 面向非独立同分布数据的自适应联邦深度学习算法. 自动化学报, 2021, x(x): 1−13 doi: 10.16383/j.aas.c201018
Zhang Ze-Hui, Li Qing-Dan, Fu Yao, He Ning-Xin, Gao Tie-Gang. Adaptive federated deep learning with Non-IID data. Acta Automatica Sinica, 2021, x(x): 1−13 doi: 10.16383/j.aas.c201018
Citation: Zhang Ze-Hui, Li Qing-Dan, Fu Yao, He Ning-Xin, Gao Tie-Gang. Adaptive federated deep learning with Non-IID data. Acta Automatica Sinica, 2021, x(x): 1−13 doi: 10.16383/j.aas.c201018

面向非独立同分布数据的自适应联邦深度学习算法

doi: 10.16383/j.aas.c201018
基金项目: 国家科技重大专项(2018YFB0204304), 天津市研究生科研创新基金(2019YJSB067)资助
详细信息
    作者简介:

    张泽辉:南开大学软件学院博士研究生. 2019年获得武汉理工大学工学硕士学位. 主要研究方向为联邦学习,,故障诊断和智能船舶控制. E-mail: zhangtianxia918@163.com

    李庆丹:南开大学软件学院硕士研究生. 主要研究方向为图像加密, 信息安全. E-mail: lqd18812745024@163.com

    富瑶:南开大学软件学院硕士研究生. 主要研究方向为云端数据完整性验证, 信息安全. E-mail: FuYao_TJ@163.com

    何宁昕:南开大学软件学院硕士研究生.2020年获得河北经贸大学工学学士学位. 主要研究方向为信息安全, 联邦学习. Email: ningxinhe1998@163.com

    高铁杠:南开大学软件学院教授, 1991获华中理工大学应用数学专业硕士学位, 2005获南开大学理学博士学位. 主要研究方向为联邦学习, 信息隐藏和云端数据安全. 本文通信作者. E-mail: gaotiegang@nankai.edu.cn

Adaptive Federated Deep Learning With Non-IID Data

Funds: Supported by National Science and Technology Major Project of China (2018YFB0204304), Tianjin Research Innovation Project for Postgraduate Students (2019YJSB067)
More Information
    Author Bio:

    ZHANG Ze-Hui Ph. D. candidate at the College of Software, Nankai University. He received his master degree from Wuhan University of Technology in 2019. His research interest covers federated learning, fault diagnosis and intelligent ship control

    LI Qing-Dan Master student at the College of Software, NanKai University. Her research interest covers image encryption and information security

    FU Yao Master student at the College of Software, NanKai University. Her research interest covers cloud data integrity verification and information security

    HE Ning-Xin Master student at the College of Software, Nankai University. She received her B.Sc degree from Hebei University of Economics and Business in 2020. Her research interest covers information security and federated learning

    GAO Tie-Gang Professor at the College of Software, Nankai University. He received his master degree in Applied Mathematics from Huazhong University of Science and Technology in 1991, and Ph.D. degree in Nankai University in 2005. His research interest covers federated learning, image watermarking, information hiding and cloud data security

  • 摘要: 近些年, 联邦学习由于能够打破数据壁垒, 实现孤岛数据价值变现, 受到了工业界和学术界的广泛关注. 然而, 在实际工程应用中, 联邦学习存在着数据隐私泄露和模型性能损失的问题. 对此, 首先对这两个问题进行了数学描述与分析. 然后, 提出了一种自适应模型聚合方案, 该方案能够设定各参与者的mini-batch值和自适应调整全局模型聚合间隔, 旨在保证模型精度的同时, 提高联邦学习训练效率. 并且, 混沌系统被首次引入联邦学习领域中, 用于构建一种基于混沌系统和同态加密的混合隐私保护方案, 从而进一步提升系统的隐私保护水平. 理论分析与实验结果表明, 提出的联邦学习算法能够保证参与者的数据隐私安全. 并且, 在非独立同分布数据的场景下, 该算法够在保证模型精度的前提下提高训练效率, 降低系统通信成本, 具备实际工业场景应用的可行性.
  • 图  1  多层神经网络模型

    Fig.  1  Multi-layer neural network model

    图  2  不同比例数据泄露的图片

    Fig.  2  Images of different proportion data leakage

    图  3  本文所提出的联邦学习系统结构图

    Fig.  3  The structure diagram of the proposed federated learning system

    图  4  联邦学习训练过程交互图

    Fig.  4  Interaction diagram of the federated learning system

    图  5  混沌加密后的参数推理实验图

    Fig.  5  Inferring data of the encrypted parameters

    图  6  在CIFAR10上不同联邦学习模型的实验曲线

    Fig.  6  Experiment curves of the different federated learning models on CIFAR10

    图  7  在F-MNIST上不同联邦学习模型的实验曲线

    Fig.  7  Experiment curves of the different federated learning models on F-MNIST

    图  8  在CIFAR-10和F-MNIST数据集的mini-batch设定消融实验曲线

    Fig.  8  Experiment curves of the minibatch size setting on CIFAR-10 and F-MNIST

    图  9  CIFAR-10自适应模型更新间隔消融实验曲线

    Fig.  9  Experiment curves of the adaptive model aggregation interval on CIFAR-10

    图  10  F-MNIST自适应模型更新间隔消融实验曲线

    Fig.  10  Experiment curves of the adaptive model aggregation interval on F-MNIST

    表  1  加密/解密算法的执行时间

    Table  1  Execution time of the encryption/decryption operations

    参数个数500200054000
    随机数生成12.05 ms25.50 ms0.40 s
    CKKS加密9.37 ms9.68 ms0.54 s
    CKKS解密1.56 ms17.18 ms0.03 s
    CKKS密文加法0.15 ms0.15 ms0.02 s
    Paillier加密3.82 s14.61 s410.32 s
    Paillier解密1.06 s4.22 s115.92 s
    Paillier密文加法7.87 ms30.03 ms0.87 s
    下载: 导出CSV

    表  2  加密/解密算法的执行次数

    Table  2  Execution numbers of the encryption/decryption operations

    模型更新间隔$\tau $5080100
    $\tau=1$[18]78001250015600
    $\tau=4$[16]155024803100
    $\tau=15$[33]5008001000
    下载: 导出CSV

    表  3  不同联邦学习方案的功能对比

    Table  3  The functionality analysis of the different FLs

    功能PFLAFLMFLAPFL
    隐私保护××
    自适应调整$\tau $××
    mini-batch设定×××
    动量项加速××
    下载: 导出CSV

    表  4  CIFAR10上不同联邦学习模型的分类结果

    Table  4  Classification results of the different federated learning models on CIFAR10

    MethodAccuracyPrecisionRecallDevavg聚合次数
    CL63.36%63.92%63.29%
    FL($\tau $=15)[33]25.76%9.34%25.87%49.91%250
    FL($\tau $=4)[16]27.64%50.14%27.76%45.04%1100
    FL($\tau $=1)[18]61.78%62.76%61.77%1.91%4400
    APFL63.66%63.49%63.64%2.02%2758
    下载: 导出CSV

    表  5  F-MNIST上不同联邦学习模型的分类结果

    Table  5  Classification results of the different federated learning models on F-MNIST

    MethodAccuracyPrecisionRecallDevavg聚合次数
    CL90.15%90.07%90.15%
    FL($\tau $=15)[33]65.99%62.18%65.99%31.43%350
    FL($\tau $=4)[16]72.77%65.24%72.77%23.16%1350
    FL($\tau $=1)[18]89.10%89.25%89.10%0.88%5250
    APFL89.36%89.30%89.36%0.87%2339
    下载: 导出CSV

    表  6  CIFAR-10下的mini-batch设定消融实验结果

    Table  6  Ablation experiment results of the mini-batch size setting on CIFAR-10

    MethodAccuracyPrecisionRecallDevavg
    CL63.36%63.92%63.29%
    FL($\tau $=15)[33]25.76%9.34%25.87%49.91%
    FL($\tau $=15)-Mbs25.70%9.14%25.78%50.07%
    FL($\tau $=4)27.64%50.14%27.76%45.04%
    FL($\tau $=4)-Mbs63.66%60.93%36.06%32.90%
    FL($\tau $=1)[18]61.78%62.76%61.77%1.91%
    FL($\tau $=1)-Mbs63.02%64.08%62.27%1.53%
    下载: 导出CSV

    表  7  F-MNIST下的mini-batch设定消融实验结果

    Table  7  Ablation experiment results of the mini-batch size setting on F-MNIST

    MethodAccuracyPrecisionRecallDevavg
    CL90.15%90.07%90.15%
    FL($\tau $=15)[33]65.99%62.18%65.99%31.43%
    FL($\tau $=15)-Mbs69.99%64.29%69.99%26.05%
    FL($\tau $=4)27.76%50.14%27.76%45.04%
    FL($\tau $=4)-Mbs76.23%84.84%76.23%14.85%
    FL($\tau $=1)[18]89.10%89.25%89.10%0.88%
    FL($\tau $=1)-Mbs89.27%89.25%89.27%0.99%
    下载: 导出CSV

    表  8  CIFAR-10下的自适应更新间隔消融实验结果

    Table  8  The ablation experiment results of the adaptive model aggregation interval on CIFAR-10

    MethodAccuracyPrecisionRecallDevavg聚合次数
    CL63.36%63.92%63.29%
    FL($\tau $ =15)[33]25.76%9.34%25.87%49.91%250
    FL($\tau $=4)[16]27.64%50.14%27.76%45.04%1100
    FL($\tau $=1)[18]61.78%62.76%61.77%1.91%4000
    APFL(no mbs)61.10%62.00%61.36%3.27%1742
    下载: 导出CSV

    表  9  F-MNIST下的自适应更新间隔消融实验结果

    Table  9  Ablation experiment results of the adaptive model aggregation interval on F-MNIST

    MethodAccuracyPrecisionRecallDevavg聚合次数
    CL90.15%90.07%90.15%
    FL($\tau $=15)[33]65.99%62.18%65.99%31.43%250
    FL($\tau $=4)[16]72.77%65.24%72.77%23.16%1100
    FL($\tau $=1)[18]89.10%89.25%89.10%0.88%4400
    APFL(no mbs)89.48%89.42%89.48%0.84%1336
    下载: 导出CSV
  • [1] 孙长银, 穆朝絮. 多智能体深度强化学习的若干关键科学问题. 自动化学报, 2020, 46(07): 1301-1312.

    Sun Chang-Yin, Mu Chao-Xu. Important scientific problems of multi-agent Deep Reinforcement Learning. Acta Automatica Sinica, 2020, 46(07): 1301-1312.
    [2] 金侠挺, 王耀南, 张辉, 等. 基于贝叶斯CNN和注意力网络的钢轨表面缺陷检测系统. 自动化学报, 2019, 45(12): 2312-2327.

    Jin Xia-Ting, Wang Yao-Nan, Zhang Hui, et al. DeepRail: automatic visual detection system for railway surface defect using bayesian CNN and attention Network. Acta Automatica Sinica, 2019, 45(12): 2312-2327.
    [3] Zhang Z, Guan C, Liu Z. Real-time optimization energy management strategy for fuel cell hybrid ships considering power sources degradation[J]. IEEE Access, 2020, 8: 87046-87059. doi: 10.1109/ACCESS.2020.2991519
    [4] Chen H, Zhang Z, Guan C, et al. Optimization of sizing and frequency control in battery/supercapacitor hybrid energy storage system for fuel cell ship. Energy, 2020, 197: 117285. doi: 10.1016/j.energy.2020.117285
    [5] 鲜征征, 李启良, 黄晓宇, 等. 基于差分隐私和SVD++的协同过滤算法. 控制与决策, 2019, 34(01): 43-54.

    Xian Zheng-Zheng, Li Qi-Liang, Huang Xiao-Yu, et al. Collaborative filtering via SVD++ with differential privacy. Control and Decision, 2019, 34(01): 43-54.
    [6] Jing L, Xk C, Sl A, et al. Privacy preservation for machine learning training and classification based on homomorphic encryption schemes. Information Sciences, 2020, 526: 166-179. doi: 10.1016/j.ins.2020.03.041
    [7] Gong M, Pan K, Xie Y, et al. Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition. Neural Networks, 2020, 125: 131-141. doi: 10.1016/j.neunet.2020.02.001
    [8] 张超, 李强, 陈子豪, 等. Medical Chain: 联盟式医疗区块链系统. 自动化学报, 2019, 45(08): 1495-1510.

    Zhang Chao, Li Qiang, Chen Zi-Hao, et al. Medical Chain: alliance medical blockchain system. Acta Automatica Sinica, 2019, 45(08): 1495-1510.
    [9] Yang Q, Liu Y, Chen T, et al. Federated machine learning: Concept and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
    [10] Li T, Sahu A K, Talwalkar A, et al. Federated learn-ing: Challenges, methods, and future directions. IEEE Signal Processing Magazine, 2020, 37(3): 50-60. doi: 10.1109/MSP.2020.2975749
    [11] Zhang W, Li X, Ma H, et al. Federated learning for machinery fault diagnosis with dynamic validation and self-supervision. Knowledge-Based Systems, 2021, 213: 106679. doi: 10.1016/j.knosys.2020.106679
    [12] Sheller M J, Edwards B, Reina G A, et al. Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific reports, 2020, 10(1): 1-12. doi: 10.1038/s41598-019-56847-4
    [13] Kwon D, Jeon J, Park S, et al. Multiagent DDPG-Based Deep Learning for Smart Ocean Federated Learning IoT Networks. IEEE Internet of Things Journal, 2020, 7(10): 9895-9903. doi: 10.1109/JIOT.2020.2988033
    [14] Rothchild D, Panda A, Ullah E, et al. Fetchsgd: Communication-efficient federated learning with sketching[C]//International Conference on Machine Learning. PMLR, 2020: 8253−8265.
    [15] Duan M, Liu D, Chen X, et al. Self-balancing federated learning with global imbalanced data in mobile systems. IEEE Transactions on Parallel and Distributed Systems, 2020, 32(1): 59-71.
    [16] Liu W, Chen L, Chen Y, et al. Accelerating Federated Learning via Momentum Gradient Descent. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(8): 1754-1766. doi: 10.1109/TPDS.2020.2975189
    [17] Wang S, Tuor T, Salonidis T, et al. Adaptive federated learning in resource constrained edge computing systems. IEEE Journal on Selected Areas in Communications, 2019, 37(6): 1205-1221. doi: 10.1109/JSAC.2019.2904348
    [18] Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345.
    [19] 张泽辉, 富瑶, 高铁杠. 支持数据隐私保护的联邦深度神经网络模型研究. 自动化学报, 2020: 1-14.

    Zhang Ze-Hui, Fu Yao, Gao Tie-Gang. Research on federated deep neural network model for data privacy protection. Acta Automatica Sinica, 2020: 1-14.
    [20] Lyu L, Li Y, Nandakumar K, et al. How to democratise and protect AI: fair and differentially private decentralised deep learning. IEEE Transactions on Dependable and Secure Computing, 2020.
    [21] Wang Y, Gu M, Ma J, et al. DNN-DP: Differential Privacy Enabled Deep Neural Network Learning Framework for Sensitive Crowdsourcing Data. IEEE Transactions on Computational Social Systems, 2019, 7(1): 215-224.
    [22] Carpov S, Gama N, Georgieva M, et al. Privacy-preserving semi-parallel logistic regression training with Fully Homomorphic Encryption. 2019: 101.
    [23] 宋蕾, 马春光, 段广晗, 等. 基于数据纵向分布的隐私保护逻辑回归. 计算机研究与发展, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414

    Song Lei, Ma Chun-Guang, Duan Guang-Han, et al. Privacy-preserving logistic regression on vertically partitioned data. Computer Research and Development, 2019, 56(10): 2243-2249. doi: 10.7544/issn1000-1239.2019.20190414
    [24] Aono Y, Hayashi T, Wang L, et al. Privacy-preserving deep learning via additively homomorphic encryption. IEEE Transactions on Information Forensics and Security, 2017, 13(5): 1333-1345.
    [25] Ou W, Zeng J, Guo Z, et al. A homomorphic-encryption-based vertical federated learning scheme for rick management. Computer Science and Information Systems, 2020, 17(3): 819-834. doi: 10.2298/CSIS190923022O
    [26] Chen H, Chillotti I, Song Y. Improved Bootstrapping for Approximate Homomorphic Encryption. Springer, Cham, 2019.
    [27] Xiao X, Wu T, Chen Y, et al. Privacy-Preserved Ap-proximate Classification Based on Homomorphic En-cryption. Mathematical and Computational Applications, 2019, 24(4): 92. doi: 10.3390/mca24040092
    [28] Zehui Z, Fu Y, Gao T. A Hybrid Image Encryption Algorithm Based on Chaos System and Simplified Ad-vanced Encryption System. International Journal of Multimedia Data Engineering and Management (IJMDEM), 2020, 11(4): 1-24. doi: 10.4018/IJMDEM.2020100101
    [29] Luo Y, Yu J, Lai W, et al. A novel chaotic image encryption algorithm based on improved baker map and logistic map. Multimedia Tools and Applications, 2019, 78(15): 22023-22043. doi: 10.1007/s11042-019-7453-3
    [30] Sathiyamurthi P, Ramakrishnan S. Speech encryption algorithm using FFT and 3D-Lorenz–logistic chaotic map. Multimedia Tools and Applications, 2020, 79(3).
    [31] Sattler F, Müller K, Samek W. Clustered federated learning: Model-agnostic distributed multitask optimization under privacy constraints. IEEE Transactions on Neural Networks and Learning Systems, 2020.
    [32] Al-Sharman M, Murdoch D, Cao D, et al. A sensor-less state estimation for a safety-oriented cyber-physical system in urban driving: deep learning approach. IEEE/CAA Journal of Automatica Sinica, 2020.
    [33] Weng J, Weng J, Zhang J, et al. Deepchain: Auditable and privacy-preserving deep learning with blockchain-based incentive. IEEE Transactions on Dependable and Secure Computing, 2019.
    [34] Sattler F, Wiedemann S, Müller K, et al. Robust and communication-efficient federated learning from non-iid data. IEEE transactions on neural networks and learning systems, 2019.
    [35] Xu G, Li H, Zhang Y, et al. Privacy-preserving federated deep learning with irregular users. IEEE Transactions on Dependable and Secure Computing, 2020.
    [36] Teng S, Wu N, Zhu H, et al. SVM-DT-based adaptive and collaborative intrusion detection. IEEE/CAA Journal of Automatica Sinica, 2017, 5(1): 108-118.
    [37] Wang F Y, Fundamental Issues in Research of Computing with Words and Linguistic Dynamic Systems. Acta Automatica Sinica (Periodical style), 2005, 31(6): 844--852.
    [38] Roychoudhury R, Bandyopadhyay S, Paul K. Adistributed mechanism for topology discovery in ad hoc wireless networks using mobile agents. In: Proceeding of IEEE First Annual Workshop on Mobile and Ad hoc Networking and Computing (Conference Proceedings style), Piscataway, USA: IEEE Press, 2000.145—146.
    [39] Hryniewicz O. An evaluation of the reliability of complex systems using shadowed sets and fuzzy lifetime data. International Journal of Automation and Computing (Periodical style—Accepted for publication), to be published.
    [40] Zhang W. Reinforcement Learning for Job-Shop Scheduling. [Ph. D. Dissertation], Oregon State University, 1996.
    [41] IEEE Criteria for Class IE Electric Systems (Standards style), IEEE Standard 308, 1969.
    [42] Jones J. Networks, 2nd ed. (Online Sources style). [Online], available: http://www.atm.com, May 10, 1991.
  • 加载中
计量
  • 文章访问数:  2190
  • HTML全文浏览量:  1457
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-12-08
  • 录用日期:  2021-03-19
  • 网络出版日期:  2021-06-04

目录

    /

    返回文章
    返回