2.765

2022影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于动态Gibbs采样的RBM训练算法研究

李飞 高晓光 万开方

李飞, 高晓光, 万开方. 基于动态Gibbs采样的RBM训练算法研究. 自动化学报, 2016, 42(6): 931-942. doi: 10.16383/j.aas.2016.c150645
引用本文: 李飞, 高晓光, 万开方. 基于动态Gibbs采样的RBM训练算法研究. 自动化学报, 2016, 42(6): 931-942. doi: 10.16383/j.aas.2016.c150645
LI Fei, GAO Xiao-Guang, WAN Kai-Fang. Research on RBM Training Algorithm with Dynamic Gibbs Sampling. ACTA AUTOMATICA SINICA, 2016, 42(6): 931-942. doi: 10.16383/j.aas.2016.c150645
Citation: LI Fei, GAO Xiao-Guang, WAN Kai-Fang. Research on RBM Training Algorithm with Dynamic Gibbs Sampling. ACTA AUTOMATICA SINICA, 2016, 42(6): 931-942. doi: 10.16383/j.aas.2016.c150645

基于动态Gibbs采样的RBM训练算法研究

doi: 10.16383/j.aas.2016.c150645
基金项目: 

国家自然科学基金 61305133,61573285

详细信息
    作者简介:

    李飞 西北工业大学电子信息学院博士研究生. 2011年获得西北工业大学系统工程专业学士学位. 主要研究方向为机器学习和深度学习. E-mail: nwpulf@mail.nwpu.edu.cn

    万开方 西北工业大学电子信息学院博士研究生. 2010年获得西北工业大学系统工程专业学士学位. 主要研究方向为航空火力控制. E-mail: yibai 2003@126.com

    通讯作者:

    高晓光 西北工业大学电子信息学院教授. 1989年获得西北工业大学飞行器导航与控制系统博士学位. 主要研究方向为贝叶斯和航空火力控制. 本文通信作者. E-mail: cxg2012@nwpu.edu.cn

Research on RBM Training Algorithm with Dynamic Gibbs Sampling

Funds: 

Supported by National Natural Science Foundation of China 61305133,61573285

More Information
    Author Bio:

    LI Fei Ph. D. candidate at the School of Electronics and Information, Northwestern Polytechnical University. He received his bachelor degree in system engineering from Northwestern Polytechnical University in 2011. His re-search interest covers machine learning and deep learning.

    WAN Kai-Fang Ph. D. candidate at the School of Electronics and In-formation, Northwestern Polytechnical University. He received his bachelor degree in system en-gineering from Northwestern Polytechnical University in 2010. His main research interest is airborne ¯re control.

    Corresponding author: GAO Xiao-Guang Professor at the School of Electronics and Informa- tion, Northwestern Polytechnical Uni- versity. She received her Ph. D. degree in aircraft naviga- tion and control system from Northwestern Polytechnical University in 1989. Her research interest covers Bayes and airborne ¯re control. Corresponding author of this paper. E-mail: cxg2012@nwpu.edu.cn
  • 摘要: 目前大部分受限玻尔兹曼机(Restricted Boltzmann machines, RBMs)训练算法都是以多步Gibbs采样为基础的采样算法. 本文针对多步Gibbs采样过程中出现的采样发散和训练速度过慢的问题,首先, 对问题进行实验描述,给出了问题的具体形式; 然后, 从马尔科夫采样的角度对多步Gibbs采样的收敛性质进行了理论分析, 证明了多步Gibbs采样在受限玻尔兹曼机训练初期较差的收敛性质是造成采样发散和训练速度过慢的主要原因; 最后, 提出了动态Gibbs采样算法,给出了对比仿真实验.实验结果表明, 动态Gibbs采样算法可以有效地克服采样发散的问题,并且能够以微小的运行时间为代价获得更高的训练精度.
  • 图  1  RBM 结构

    Fig.  1  Con¯guration of RBM

    图  2  原始数据灰度图

    Fig.  2  Gray image of initial data

    图  3  重构误差图

    Fig.  3  Reconstruction error diagram

    图  4  运行时间图

    Fig.  4  Runtime diagram

    图  5  CD_1 采样灰度图

    Fig.  5  Gray image of CD_1 sampling

    图  6  CD_5 采样灰度图

    Fig.  6  Gray image of CD_5 sampling

    图  7  CD_10 采样灰度图

    Fig.  7  Gray image of CD_10 sampling

    图  8  CD_100 采样灰度图

    Fig.  8  Gray image of CD_100 sampling

    图  9  CD_500 采样灰度图

    Fig.  9  Gray image of CD_500 sampling

    图  10  CD_1000 采样灰度图

    Fig.  10  Gray image of CD_1000 sampling

    图  11  CD_500 采样灰度图

    Fig.  11  Gray image of CD_500 sampling

    图  12  CD_1000 采样灰度图

    Fig.  12  Gray image of CD_1000 sampling

    图  13  CD_500 采样灰度图

    Fig.  13  Gray image of CD_500 sampling

    图  14  CD_1000 采样灰度图

    Fig.  14  Gray image of CD_1000 sampling

    图  15  采样误差局部放大图

    Fig.  15  Local enlarged drawing of reconstruction error in initial phase

    图  16  CD_1000 采样误差局部放大图

    Fig.  16  Local enlarged drawing of reconstruction error in later stage

    图  17  重构误差对比图

    Fig.  17  Contrast of reconstruction error

    图  18  训练初期局部放大图

    Fig.  18  Local enlarged drawing of reconstruction error in initial phase

    图  19  训练后期局部放大图

    Fig.  19  Local enlarged drawing of reconstruction error in later stage

    图  20  运行时间对比图

    Fig.  20  Contrast of runtime

    图  21  运行时间对比图

    Fig.  21  Contrast of runtime

    图  22  运行时间对比图

    Fig.  22  Contrast of runtime

    图  23  运行时间对比图

    Fig.  23  Contrast of runtime

    图  24  DGS 迭代10次采样灰度图

    Fig.  24  Gray image of DGS by 10 iterations

    图  25  DGS 迭代20次采样灰度图

    Fig.  25  Gray image of DGS by 20 iterations

    图  26  DGS 迭代30次采样灰度图

    Fig.  26  Gray image of DGS by 30 iterations

    图  27  DGS 迭代40次采样灰度图

    Fig.  27  Gray image of DGS by 40 iterations

    图  28  DGS 迭代50次采样灰度图

    Fig.  28  Gray image of DGS by 50 iterations

    图  29  DGS 重构灰度图

    Fig.  29  Gray image of DGS

    表  1  网络参数初值

    Table  1  Initial value of parameters

    网络参数初始值
    a zeros(1, 784)
    b zeros(1, 500)
    w 0.1 × randn(784, 500)
    η 0.1
    下载: 导出CSV

    表  2  实验分组

    Table  2  Experimental grouping

    数据集算法迭代次数
    MNIST CD_11000
    MNIST CD_51000
    MNIST CD_101000
    MNIST CD_1001000
    MNIST CD_5001000
    MNIST CD_10001000
    下载: 导出CSV

    表  3  实验分组

    Table  3  Experimental grouping

    数据集训练算法Iter
    MNIST CD_11000
    MNIST CD_51000
    MNIST CD_101000
    MNIST CD_1001000
    MNIST CD_5001000
    MNIST CD_10001000
    MNIST DGS1000
    下载: 导出CSV

    表  4  网络参数初值

    Table  4  Initial values of parameters

    算法参数CD_k DGS
    a zeros(1,784) zeros(1,784)
    b zeros(1,500) zeros(1,500)
    w 0.1 × randn(784,500) 0.1 × randn(784,500)
    η0.10.1
    V784784
    H500500
    下载: 导出CSV

    表  5  DGS 迭代策略

    Table  5  Iterative strategy of DGS

    M Gibbs_N
    (1:m1) = (1:300) Gibbs_N1 = 1
    (m1:m2) = (300:900) Gibbs_N2 = 5
    (m2:Iter) = (900:1000) Gibbs_N3 = 10
    下载: 导出CSV
  • [1] Hinton G E, Salakhutdinov R R. Reducing the dimensionality of data with neural networks. Science, 2006, 313(5786): 504-507
    [2] Le Roux N, Heess N, Shotton J, Winn J. Learning a generative model of images by factoring appearance and shape. Neural Computation, 2011, 23(3): 593-650
    [3] 苏连成,朱枫. 一种新的全向立体视觉系统的设计. 自动化学报, 2006, 32(1): 67-72

    Su Lian-Cheng, Zhu Feng. Design of a novel omnidirectional stereo vision system. Acta Automatica Sinica, 2006, 32(1): 67-72
    [4] Bengio Y. Learning deep architectures for AI. Foundations and Trends® in Machine Learning, 2009, 2(1): 1-127
    [5] Deng L, Abdel-Hamid O, Yu D. A deep convolutional neural network using heterogeneous pooling for trading acoustic invariance with phonetic confusion. In: Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Vancouver, BC: IEEE, 2013. 6669-6673
    [6] Deng L. Design and learning of output representations for speech recognition. In: Proceedings of the Neural Information Processing Systems (NIPS) Workshop on Learning Output Representations[Online], available: http://research.microsoft.com/apps/pubs/default.aspx?id=204702, July 14, 2015
    [7] Chet C C, Eswaran C. Reconstruction and recognition of face and digit images using autoencoders. Neural Computing and Applications, 2010, 19(7): 1069-1079
    [8] Deng L, Hinton G, Kingsbury B. New types of deep neural network learning for speech recognition and related applications: an overview. In: Proceedings of the 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Vancouver, BC: IEEE, 2013. 8599-8603
    [9] Erhan D, Courville A, Bengio Y, Vincent P. Why does unsupervised pre-training help deep learning? In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010). Sardinia, Italy, 2010. 201-208
    [10] Salakhutdinov R, Hinton G. Deep Boltzmann machines. In: Proceedings of the 12th International Conference on Artificial Intelligence and Statistics (AISTATS 2009). Florida, USA, 2009. 448-455
    [11] Swersky K, Chen B, Marlin B, de Freitas N. A tutorial on stochastic approximation algorithms for training restricted Boltzmann machines and deep belief nets. In: Proceedings of the 2010 Information Theory and Applications Workshop (ITA). San Diego, CA: IEEE, 2010. 1-10
    [12] Hinton G E, Osindero S, Teh Y W. A fast learning algorithm for deep belief nets. Neural Computation, 2006, 18(7): 1527-1554
    [13] Fischer A, Igel C. Bounding the bias of contrastive divergence learning. Neural Computation, 2011, 23(3): 664-673
    [14] Tieleman T. Training restricted Boltzmann machines using approximations to the likelihood gradient. In: Proceedings of the 25th International Conference on Machine Learning (ICML). New York: ACM, 2008. 1064-1071
    [15] Tieleman T, Hinton G E. Using fast weights to improve persistent contrastive divergence. In: Proceedings of the 26th Annual International Conference on Machine Learning (ICML). New York: ACM, 2009. 1033-1040
    [16] Sutskever I, Tieleman T. On the convergence properties of contrastive divergence. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010). Sardinia, Italy, 2010. 789-795
    [17] Fischer A, Igel C. Parallel tempering, importance sampling, and restricted Boltzmann machines. In: Proceedings of 5th Workshop on Theory of Randomized Search Heuristics (ThRaSH),[Online], available: http://www.imm.dtu.dk/projects/thrash-workshop/schedule.php, August 20, 2015
    [18] Desjardins G, Courville A, Bengio Y. Adaptive parallel tempering for stochastic maximum likelihood learning of RBMs. In: Proceedings of NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning. Granada, Spain, 2010.
    [19] Cho K, Raiko T, Ilin A. Parallel tempering is efficient for learning restricted Boltzmann machines. In: Proceedings of the WCCI 2010 IEEE World Congress on Computational Intelligence. Barcelona, Spain: IEEE, 2010. 3246-3253
    [20] Brakel P, Dieleman S, Schrauwen B. Training restricted Boltzmann machines with multi-tempering: harnessing parallelization. In: Proceedings of the 22nd International Conference on Artificial Neural Networks. Lausanne, Switzerland: Springer, 2012. 92-99
    [21] Desjardins G, Courville A, Bengio Y, Vincent P, Delalleau O. Tempered Markov chain Monte Carlo for training of restricted Boltzmann machines. In: Proceedings of the 13th International Conference on Artificial Intelligence and Statistics (AISTATS 2010). Sardinia, Italy, 2010. 145-152
    [22] Fischer A, Igel C. Training restricted Boltzmann machines: an introduction. Pattern Recognition, 2014, 47(1): 25-39
    [23] Hinton G E. A practical guide to training restricted Boltzmann machines. Neural Networks: Tricks of the Trade (2nd Edition). Berlin Heidelberg: Springer, 2012. 599-619
  • 加载中
图(29) / 表(5)
计量
  • 文章访问数:  2037
  • HTML全文浏览量:  389
  • PDF下载量:  1540
  • 被引次数: 0
出版历程
  • 收稿日期:  2015-10-19
  • 录用日期:  2016-05-03
  • 刊出日期:  2016-06-20

目录

    /

    返回文章
    返回