Parameter Optimization of Leaky Integrator Echo State Network with Internal-point Penalty Function Method
-
摘要: 为了提升泄露积分型回声状态网（Leaky integrator echo state network，Leaky-ESN）的性能，提出利用罚函数内点法优化Leaky-ESN的全局参数，如泄漏率、内部连接权矩阵谱半径、输入比例因子等，这克服了通过反复试验法选取参数值而降低了Leaky-ESN模型的优越性和性能.Leaky-ESN的全局参数必须保障回声状态网满足回声状态特性，因此它们之间存在不等式约束条件.有学者提出利用随机梯度下降法来优化内部连接权矩阵谱半径、输入比例因子、泄露率三个全局参数，一定程度上提高了Leaky-ESN的逼近精度.然而，随机梯度下降法是解决无约束优化问题的基本算法，在利用随机梯度下降法优化参数时，没有考虑参数必须满足回声特性的约束条件（不等式约束条件），致使得到的参数值不是最优解.由于罚函数内点法可以求解具有不等式约束的最优化问题，应用范围广，收敛速度较快，具有很强的全局寻优能力.因此，本文提出利用罚函数内点法优化Leaky-ESN的全局参数，并以时间序列预测为例，检验优化后的Leaky-ESN的预测性能，仿真结果表明了本文提出方法的有效性.Abstract: To improve leaky integrator echo state network (Leaky-ESN) performance, internal-point penalty function (IPF) method is used to optimize the global parameters of Leaky-ESN, such as leakage rate, spectral radius of internal connection weight matrix, scaling of input, etc., which overcomes loss of superiority and performance of Leaky-ESN because of using trial and error method to select parameter values. The global parameters of Leaky-ESN have to guarantee the echo state network to meet the echo state property, thus inequality constraints exist between them. Some researchers put forward the method using the stochastic gradient descent (GD) to optimize leakage rate, spectral radius of internal connection weight matrix, and scaling of input, which can improve the approximation precision of the Leaky-ESN to some certain extent. However, the stochastic gradient descent method is a basic algorithm to solve unconstrained optimization problems. Without considering parameters which need satisfy the constraint conditions of the echo state property (inequality constraints) during using stochastic gradient descent method, the parameter value is not the optimal solution. Internal-point penalty function method can solve the optimized problem with inequality constraints, a wide scope of application, fast convergence speed, strong ability of global optimization. Therefore, in this paper, internal-point penalty function method is used to optimize the global parameters of Leaky-ESN, and time series prediction is selected as an example to examine the performance of the optimized Leaky-ESN. Simulation results show the effectiveness of the proposed approach.1) 本文责任编委 魏庆来
-
表 1 Leaky-ESN的测试NRMSE
Table 1 Testing NRMSE of Leaky-ESN
方法 第一时间序列 第二时间序列 随机梯度下降法 0.21240 0.17710 罚函数内点法 0.08698 0.01397 
下载: 导出CSV
-
[1] Jaeger H. Tutorial on Training Recurrent Neural Networks, Covering BPTT, RTRL, EKF, and the "Echo State Network" Approach. Technical Report GMD Report 159, German National Research Center for Information Technology, German, 2002. [2] Jaeger H. The "Echo State" Approach to Analysing and Training Recurrent Neural Networks. Technical Report GMD report 148, German National Research Center for Information Technology, German, 2001. [3] Jaeger H, Haas H. Harnessing nonlinearity:predicting chaotic systems and saving energy in wireless communication. Science, 2004, 304(5667):78-80 doi: 10.1126/science.1091277 [4] Lun S X, Wang S, Guo T T, Du C J. An Ⅰ-Ⅴ model based on time warp invariant echo state network for photovoltaic array with shaded solar cells. Solar Energy, 2014, 105:529-541 doi: 10.1016/j.solener.2014.04.023 [5] Skowronski M D, Harris J G. Noise-robust automatic speech recognition using a predictive echo state network. IEEE Transactions on Audio, Speech, and Language Processing, 2007, 15(5):1724-1730 doi: 10.1109/TASL.2007.896669 [6] Han S I, Lee J M. Precise positioning of nonsmooth dynamic systems using fuzzy wavelet echo state networks and dynamic surface sliding mode control. IEEE Transactions on Industrial Electronics, 2013, 60(11):5124-5136 doi: 10.1109/TIE.2012.2218560 [7] Li G Q, Niu P F, Zhang W P, Zhang Y. Control of discrete chaotic systems based on echo state network modeling with an adaptive noise canceler. Knowledge-Based Systems, 2012, 35(15):35-40 http://www.sciencedirect.com/science/article/pii/S0950705112001153 [8] Song R Z, Xiao W D, Sun C Y. A new self-learning optimal control laws for a class of discrete-time nonlinear systems based on ESN architecture. Science China Information Sciences, 2014, 57(6):Article No. 068202 doi: 10.1007%2Fs11432-013-4954-y.pdf [9] Lun S X, Yao X S, Qi H Y, Hu H F. A novel model of leaky integrator echo state network for time-series prediction. Neurocomputing, 2015, 159(1):58-66 http://linkinghub.elsevier.com/retrieve/pii/S0925231215001782 [10] Bianchi F M, Scardapane S, Uncini A, Rizzi A, Sadeghian A. Prediction of telephone calls load using echo state network with exogenous variables. Neural Networks, 2015, 71(C):204-213 https://www.researchgate.net/publication/281677244_Prediction_of_telephone_calls_load_using_Echo_State_Network_with_exogenous_variables [11] Jaeger H, Lukoševičius M, Popovici D, Siewert U. Optimization and applications of echo state networks with leaky-integrator neurons. Neural Networks, 2007, 20(3):335-352 doi: 10.1016/j.neunet.2007.04.016 [12] Lukoševičius M. A practical guide to applying echo state networks. Neural Networks:Tricks of the Trade (Second Edition). Berlin Heidelberg:Springer-Verlag, 2012. 659-686 [13] Nocedal J, Wright S J. Numerical Optimization (Second Edition). New York:Springer, 2006. 30-31 -
图(9) / 表(1)
计量
- 文章访问数: 1817
- HTML全文浏览量: 134
- PDF下载量: 721
- 被引次数: 0
