-
摘要: 人们对大数据的认识已从"3Vs" (Volume-大容量; Variety-多样性; Velocity-处理实时性)、"4Vs" ("3Vs"与Value-价值)、到现今的"5Vs" ("4Vs"与Veracity-真实性).在此背景下, 首先分析过程工业大数据的"5Vs"特性; 接下来, 综述现有数据建模方法, 并结合过程工业大数据特有性质 (包括:多层面不规则采样性、多时空时间序列性、不真实数据混杂性) 论述现有数据建模方法应用于工业大数据建模时的局限; 最后, 探讨过程工业大数据建模有待研究的问题, 包括:1) 多层面不规则采样数据的潜结构建模; 2) 用于事件发现、决策和因果分析的多时空时间序列数据建模; 3) 含有不真实数据的鲁棒建模; 4) 支持实时建模的大容量数据计算架构与方法.
-
关键词:
- 过程工业大数据 /
- 多层面数据潜结构建模 /
- 多时空时间序列数据建模 /
- 大数据计算架构
Abstract: The understanding of big data goes through three stages, i.e., "3Vs" (Volume, variety and velocity), "4Vs" ("3Vs" and value), and "5Vs" ("4Vs" and veracity). In the era of big data of process industries, the "5Vs" characteristics of industrial big data are analyzed. After that, the existing methods on data modeling are reviewed while the corresponding limitations are analyzed under industrial big data circumstances with specific characteristics, i.e., multi-layer irregularly sampling, multiple temporal and spatial time series, and non-veracity with outlier. Finally, the perspectives on industrial big data modeling are discussed, including: i) latent structure modeling of multi-layer irregularly sampled big data; ii) multiple temporal and spatial time-series data modeling for event discovery, decision-making, and causality analysis; iii) robust modeling of data with non-veracity samples; and iv) data-friendly system architecture and method towards big data real-time modeling. -
[1] Wu X D, Zhu X Q, Wu G Q, Ding W. Data mining with big data. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(1):97-107 doi: 10.1109/TKDE.2013.109 [2] Syed A R, Gillela K, Venugopal C. The future revolution on big data. International Journal of Advanced Research in Computer and Communication Engineering, 2013, 2(6):2446-2451 http://www.doc88.com/p-6631367365432.html [3] Condliffe J. The problem with big data is that nobody und-erstands it[Online], available:http://gizmodo.com/59062-04/the-problem-with-big-data-is-that-nobody-understan-ds-it, April 30, 2012. [4] Manyika J, Chui M, Brown B, Bughin J, Dobbs R, Roxburgh C, Byers A H. Big data:the next frontier for innovation, competition, and productivity. McKinsey Global Institute Report[Online], available:http://www.mckinsey.com/insights/mgi/research/technology_and_innovation/big_data_the_next_frontier_for_innovation, June, 2011. [5] Halevi G, Moed H. The Evolution of big data as a research and scientific topic:overview of the literature. Special Issue on Big Data, Research Trends, 2012, (30):1-37 [6] Ginsberg J, Mohebbi M H, Patel R S, Brammer L, Smolinski M S, Brilliant L. Detecting influenza epidemics using search engine query data. Nature, 2009, 457(7232):1012-1014 doi: 10.1038/nature07634 [7] Preis T, Moat H S, Stanley H E. Quantifying trading behavior in financial markets using Google trends. Scientific Reports, 2013, 3:1684 http://adsabs.harvard.edu/abs/2013NatSR...3E1684P [8] GE智能平台.工业大数据云利用大数据集推动创新、竞争和增长.自动化博览, 2012, (12):40-42 http://www.cnki.com.cn/Article/CJFDTOTAL-ZDBN201212025.htmGE intelligent platform. Industrial big data cloud promotes innovation, competition and growth using big data. Automation Panorama, 2012, (12):40-42 http://www.cnki.com.cn/Article/CJFDTOTAL-ZDBN201212025.htm [9] 钟路音.工业数据增速是其他大数据领域的两倍.人民邮电报.)[Online], available:http://www.cnii.com.cn/wlkb/rmydb/content/2013-08/27/content_1210645.htm, August 27, 2013.Zhong Lu-Yin. Industrial data growth rate will be two times the other big data fields. People's Posts and Telecommunications News. [10] Industrial Big Data. Know the future-automate processes. Software for data analysis and accurate forecasting[Online], available:http://differentia.co/qlikview/docs/Blue-Yonder-White-Paper-Industrial-Big-Data.pdf, October 23, 2015.. [11] Obitko M, Jirkovský V, Bezdíček J. Big data challenges in industrial automation. Industrial Applications of Holonic and Multi-Agent Systems, Lecture Notes in Computer Science. Berlin Heidelberg:Springer, 2013, 8062:305-316 http://www.springer.com/gp/book/9783642400896 [12] Schroeck M, Shockley R, Smart J, Romero-Mora-les D, Tu-fano P. Analytics:the real-world use of big data[Online], available:http://www-03.ibm.com/systems/hu/resources/the_real_word_use_of_big_data.pdf, 2013. [13] Hillard R. It's time for a new definition of big data[Online], available:http://mike2.openmethodology.org/blogs/information-development/2012/03/18/its-time-for-a-new-defin-ition-of-big-data, March 18, 2012. [14] Yan J. Big data, bigger opportunities[Online], available:http://www.meritalk.com/pdfs/bdx/bdx-whitepaper-090413.pdf, April 9, 2013. [15] Qin S J. Survey on data-driven industrial process monitoring and diagnosis. Annual Reviews in Control, 2012, 36(2):220-234 doi: 10.1016/j.arcontrol.2012.09.004 [16] Kano M, Tanaka S, Hasebe S, Hashimoto I, Ohno H. Monitoring independent components for fault detection. AIChE Journal, 2003, 49(4):969-976 doi: 10.1002/(ISSN)1547-5905 [17] Lee J M, Qin S J, Lee I B. Fault detection and diagnosis based on modified independent component analysis. AIChE Journal, 2006, 52(10):3501-3514 doi: 10.1002/(ISSN)1547-5905 [18] Ku W F, Storer R H, Georgakis C. Disturbance detection and isolation by dynamic principal component analysis. Chemometrics and Intelligent Laboratory Systems, 1995, 30(1):179-196 doi: 10.1016/0169-7439(95)00076-3 [19] Singhal A, Seborg D E. Evaluation of a pattern matching method for the Tennessee Eastman challenge process. Journal of Process Control, 2006, 16(6):601-613 doi: 10.1016/j.jprocont.2005.10.005 [20] Yoo C K, Villez K, Lee I B, Rosén C, Vanrolleghem P A. Multi-model statistical process monitoring and diagnosis of a sequencing batch reactor. Biotechnology and Bioengineering, 2007, 96(4):687-701 doi: 10.1002/(ISSN)1097-0290 [21] Kano M, Hasebe S, Hashimoto I, Ohno H. Evolution of multivariate statistical process control:application of independent component analysis and external analysis. Computers & Chemical Engineering, 2004, 28(6-7):1157-1166 https://www.researchgate.net/publication/222649593_Evolution_of_multivariate_statistical_process_control_Application_of_independent_component_analysis_and_external_analysis [22] Rosipal R. Kernel partial least squares for nonlinear regression and discrimination. Neural Network World, 2003, 13(3):291-300 https://www.researchgate.net/profile/Roman_Rosipal/publication/228639853_Kernel_Partial_Least_Squares_for_Nonlinear_Regression_and_Discrimination/links/02e7e537efbddccf5e000000/Kernel-Partial-Least-Squares-for-Nonlinear-Regression-and-Discrimination.pdf [23] Sheng N, Liu Q, Qin S J, Chai T Y. Comprehensive monitoring of nonlinear processes based on concurrent kernel projection to latent structures. IEEE Transactions on Automation Science and Engineering, 2015, (99):1-9, DOI: 10.1109/TASE.2015.2477272 [24] Negiz A, Çinar A. Statistical monitoring of multivariable dynamic processes with state-space models. AIChE Journal, 1997, 43(8):2002-2020 doi: 10.1002/(ISSN)1547-5905 [25] Simoglou A, Martin E B, Morris A J. Statistical performance monitoring of dynamic multivariate processes using state space modelling. Computers & Chemical Engineering, 2002, 26(6):909-920 https://www.researchgate.net/publication/223030941_Statistical_performance_monitoring_of_dynamic_multivariate_processes_using_state_space_modeling [26] Qin S J. An overview of subspace identification. Computers & Chemical Engineering, 2006, 30(10-12):1502-1513 https://www.researchgate.net/publication/223164646_An_overview_of_subspace_identification [27] Wang J, Qin S J. A new subspace identification approach based on principal component analysis. Journal of Process Control, 2002, 12(8):841-855 doi: 10.1016/S0959-1524(02)00016-1 [28] Li W H, Qin S J. Consistent dynamic PCA based on errors-in-variables subspace identification. Journal of Process Control, 2001, 11(6):661-678 doi: 10.1016/S0959-1524(00)00041-X [29] Ding S X, Zhang P, Naik A, Ding E L, Huang B. Subspace method aided data-driven design of fault detection and isolation systems. Journal of Process Control, 2009, 19(9):1496-1510 doi: 10.1016/j.jprocont.2009.07.005 [30] Wen Q J, Ge Z Q, Song Z H. Data-based linear Gaussian state-space model for dynamic process monitoring. AIChE Journal, 2012, 58(12):3763-3776 doi: 10.1002/aic.13776 [31] Li G, Qin S J, Zhou D H. A new method of dynamic latent variable modeling for process monitoring. IEEE Transactions on Industrial Electronics, 2014, 61(11):6438-6445 doi: 10.1109/TIE.2014.2301761 [32] Kaspar M H, Ray W H. Dynamic PLS modelling for process control. Chemical Engineering Science, 1993, 48(20):3447-3461 doi: 10.1016/0009-2509(93)85001-6 [33] Dong Y N, Qin S J. Dynamic-inner partial least squares for dynamic data modeling. In:Proceedings of the 9th International Symposium on Advanced Control of Chemical Processes (ADCHEM). Whistler, British Columbia, Canada:IFAC, 2015. 117-122 [34] Li G, Liu B S, Qin S J, Zhou D H. Quality relevant data-driven modeling and monitoring of multivariate dynamic processes:the dynamic T-PLS approach. IEEE Transactions on Neural Networks, 2011, 22(12):2262-2271 doi: 10.1109/TNN.2011.2165853 [35] Liu Q, Qin S J, Chai T Y. Quality-relevant monitoring and diagnosis with dynamic comcurrent projection to latent structures. In:Proceedings of the 19th IFAC World Congress. Cape Town, South Africa:IFAC, 2014. 2740-2745 [36] Dean J, Ghemawat S. MapReduce:simplified data processing on large clusters. In:Proceedings of the 6th Symposium on Operating Systems Design and Implementation. Berkeley, CA, USA:USENIX Association, 2004. 137-149 [37] Dittrich J, Quiané-Ruiz J A, Jndal A, Kargin Y, Setty V, Schad J. Hadoop++:making a yellow elephant run like a cheetah (without it even noticing). Proceedings of the VLDB Endowment, 2010, 3(1-2):515-529 doi: 10.14778/1920841 [38] Su X Y, Swart G. Oracle in-database hadoop:when mapreduce meets RDBMS. In:Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2012. 779-790 [39] Silva Y A, Reed J M. Exploiting MapReduce-based similarity joins. In:Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2012. 693-696 [40] Gudmundsson G P, Amsaleg L, Jonsson B P. Distributed high-dimensional index creation using Hadoop, HDFS and C++. In:Proceedings of the 10th International Workshop on Content-Based Multimedia Indexing. Annecy, France:IEEE, 2012. 1-6 [41] Yang L, Shi Z Z, Xu L D, Liang F, Kirsh I. DH-TRIE frequent pattern mining on Hadoop using JPA. In:Proceedings of the 2011 IEEE International Conference on Granular Computing. Kaohsiung:IEEE, 2011. 875-878 [42] Böse J H, Andrzejak A, Högqvist M. Beyond online aggregation:parallel and incremental data mining with online Map-Reduce. In:Proceedings of the 2010 Workshop on Massive Data Analytics on the Cloud. New York, USA:ACM, 2010. Article No. 3 [43] Condie T, Conway N, Alvaro P, Hellerstein J M, Elmeleegy K, Sears R. MapReduce online. In:Proceedings of the 7th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA, USA:USENIX Association, 2010. 313-328 [44] Bu Y Y, Howe B, Balazinska M, Ernst M D. HaLoop:efficient iterative data processing on large clusters. Proceedings of the VLDB Endowment, 2010, 3(1-2):285-296 doi: 10.14778/1920841 [45] Zhang Y F, Gao Q X, Gao L X, Wang C R. iMapReduce:a distributed computing framework for iterative computation. Journal of Grid Computing, 2012, 10(1):47-68 doi: 10.1007/s10723-012-9204-9 [46] Elnikety E, Elsayed T, Ramadan H E. iHadoop:asynchronous iterations for MapReduce. In:Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science. Athens:IEEE, 2011. 81-90 [47] Zhang Y F, Gao Q X, Gao L X, Wang C R. PrIter:a distributed framework for prioritizing iterative computations. IEEE Transactions on Parallel and Distributed Systems, 2013, 24(9):1884-1893 doi: 10.1109/TPDS.2012.272 [48] Mazur E, Li B D, Diao Y L, Shenoy P J. Towards scalable one-pass analytics using MapReduce. In:Proceedings of the 2011 IEEE International Symposium on Parallel and Distributed Processing Workshops and Ph.D Forum. Shanghai, China:IEEE, 2011. 1102-1111 [49] Li B D, Mazur E, Diao Y L, McGreor A, Shenoy P. A platform for scalable one-pass analytics using MapReduce. In:Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2011. 985-996 [50] Brito A, Martin A, Knauth T, Creutz S, Becker D, Weigert S, Fetzer C. Scalable and low-latency data processing with stream MapReduce. In:Proceedings of the 2011 IEEE 3rd International Conference on Cloud Computing Technology and Science. Athens:IEEE, 2011. 48-58 [51] Sato-Ilic M. Preface to Part Ⅲ Adaptive big data analytics. Procedia Computer Science, 2012, 12:211 doi: 10.1016/j.procs.2012.09.057 [52] Yan W Z, Brahmakshatriya U, Xue Y, Gilder M, Wise B. p-PIC:parallel power iteration clustering for big data. Journal of Parallel and Distributed Computing, 2013, 73(3):352-359 doi: 10.1016/j.jpdc.2012.06.009 [53] Zhao W Z, Ma H F, He Q. Parallel K-means clustering based on MapReduce. Cloud Computing, 2009, 5931:674-679 doi: 10.1007/978-3-642-10665-1 [54] Gao H, Jiang J, She L, Fu Y. A new agglomerative hierarchical clustering algorithm implementation based on the map reduce framework. International Journal of Digital Content Technology and Its Applications, 2010, 4(3):95-100 doi: 10.4156/jdcta [55] Ordonez C, Pitchaimalai S K. Fast UDFs to compute sufficient statistics on large data sets exploiting caching and sampling. Data & Knowledge Engineering, 2010, 69(4):383-398 https://www.researchgate.net/publication/223782408_Fast_UDFs_to_compute_sufficient_statistics_on_large_data_sets_exploiting_caching_and_sampling [56] Qin S J. Process data analytics in the era of big data. AIChE Journal, 2014, 60(9):3092-3100 doi: 10.1002/aic.v60.9 [57] Alma Ö G. Comparison of robust regression methods in linear regression. International Journal of Contemporary Mathematical Sciences, 2011, 6(9):409-421 https://www.researchgate.net/publication/229017751_Comparison_of_Robust_Regression_Methods_in_Linear_Regression [58] 周晓剑.考虑梯度信息的ε-支持向量回归机.自动化学报, 2014, 40(12):2908-2915 http://www.aas.net.cn/CN/abstract/abstract18568.shtmlZhou Xiao-Jian. Enhancing ε-support vector regression with gradient information. Acta Automatica Sinica, 2014, 40(12):2908-2915 http://www.aas.net.cn/CN/abstract/abstract18568.shtml [59] 曹鹏飞, 罗雄麟.基于Wiener结构的软测量模型及辨识算法.自动化学报, 2014, 40(10):2179-2192 http://www.aas.net.cn/CN/abstract/abstract18493.shtmlCao Peng-Fei, Luo Xiong-Lin. Wiener structure based modeling and identifying of soft sensor systems. Acta Automatica Sinica, 2014, 40(10):2179-2192 http://www.aas.net.cn/CN/abstract/abstract18493.shtml [60] 钱富才, 黄姣茹, 秦新强.基于鲁棒优化的系统辨识算法研究.自动化学报, 2014, 40(5):988-993 http://www.aas.net.cn/CN/abstract/abstract18368.shtmlQian Fu-Cai, Huang Jiao-Ru, Qin Xin-Qiang. Research on algorithm for system identification based on robust optimization. Acta Automatica Sinica, 2014, 40(5):988-993 http://www.aas.net.cn/CN/abstract/abstract18368.shtml [61] Trygg J, Wold S. Orthogonal projections to latent structures (O-PLS). Journal of Chemometrics, 2002, 16(3):119-128 doi: 10.1002/(ISSN)1099-128X [62] Li G, Qin S J, Zhou D H. Output relevant fault reconstruction and fault subspace extraction in total projection to latent structures models. Industrial & Engineering Chemistry Research, 2010, 49(19):9175-9183 https://www.researchgate.net/publication/231391323_Output_Relevant_Fault_Reconstruction_and_Fault_Subspace_Extraction_in_Total_Projection_to_Latent_Structures_Models [63] Zhou D H, Li G, Qin S J. Total projection to latent structures for process monitoring. AIChE Journal, 2010, 56(1):168-178 https://www.researchgate.net/publication/229883646_Total_projection_to_latent_structures_for_process_monitoring [64] Qin S J, Zheng Y Y. Quality-relevant and process-relevant fault monitoring with concurrent projection to latent structures. AIChE Journal, 2013, 59(2):496-504 doi: 10.1002/aic.v59.2 [65] Liu Q, Qin S J, Chai T Y. Multiblock concurrent PLS for decentralized monitoring of continuous annealing processes. IEEE Transactions on Industrial Electronics, 2014, 61(11):6429-6437 doi: 10.1109/TIE.2014.2303781 [66] Bengio Y. Learning deep architectures for AI. Foundations and Trends in Machine Learning, 2009, 2(1):1-127 doi: 10.1561/2200000006 [67] Qin S J. Process monitoring in the era of big data. In:Proceeding of the 9th International Symposium on Advanced Control of Chemical Processes (ADCHEM). Plenary Talk, Whictler, British Columbia, Canada:IFAC, 2015. [68] Fu T C. A review on time series data mining. Engineering Applications of Artificial Intelligence, 2011, 24(1):164-181 doi: 10.1016/j.engappai.2010.09.007 [69] Keogh E, Kasetty S. On the need for time series data mining benchmarks:a survey and empirical demonstration. Data Mining and Knowledge Discovery, 2003, 7(4):349-371 doi: 10.1023/A:1024988512476 [70] Rakthanmanon T, Campana B, Mueen A, Batista G, Westover B, Westover B, Zhu Q, Zakaria J, Keogh E. Addressing big data time series:mining trillions of time series subsequences under dynamic time warping. ACM Transactions on Knowledge Discovery from Data, 2013, 7(3):Article No.10 https://www.researchgate.net/publication/262394370_Addressing_Big_Data_Time_Series_Mining_Trillions_of_Time_Series_Subsequences_Under_Dynamic_Time_Warping [71] Jegou H, Douze M, Schmid C, Perez P. Aggregating local descriptors into a compact image representation. In:Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition. San Francisco, CA:IEEE, 2010. 3304-3311 [72] Yuan T, Qin S J. Root cause diagnosis of plant-wide oscillations using Granger causality. Journal of Process Control, 2014, 24(2):450-459 doi: 10.1016/j.jprocont.2013.11.009 [73] Kadlec P, Gabrys B, Strandt S. Data-driven soft sensors in the process industry. Computers & Chemical Engineering, 2009, 33(4):795-814 http://www.sciencedirect.com/science/article/pii/S0098135409000076 [74] Suykens J A K, de Brabanter J, Lukas L, Vandewalle J. Weighted least squares support vector machines:robustness and sparse approximation. Neurocomputing, 2002, 48(1-4):85-105 doi: 10.1016/S0925-2312(01)00644-0 [75] 张淑宁, 王福利, 何大阔, 贾润达.在线鲁棒最小二乘支持向量机回归建模.控制理论与应用, 2011, 28(11):1601-1606 http://www.cnki.com.cn/Article/CJFDTOTAL-KZLY201111012.htmZhang Shu-Ning, Wang Fu-Li, He Da-Kuo, Jia Run-Da. Modeling method of online robust least-squares-support-vector regression. Control Theory & Applications, 2011, 28(11):1601-1606 http://www.cnki.com.cn/Article/CJFDTOTAL-KZLY201111012.htm [76] Mackey L W, Talwalkar A, Jordan M I. Divide-and-conquer matrix factorization. Advances in Neural Information Processing Systems, 2011, 24:1134-1142 http://bigdata2013.sciencesconf.org/conference/bigdata2013/pages/mackey.pdf [77] Candés E J, Li X D, Ma Y, Wright J. Robust principal component analysis. Journal of the ACM, 2011, 58(3):Article No.11 http://lagrange.math.siu.edu/Olive/pprpca.pdf [78] Mahoney M W. Randomized algorithms for matrices and data. Foundations and Trends in Machine Learning, 2011, 3(2):123-224 http://www.sfbayacm.org/sites/default/files/rafmad1.pdf -
下载:
图(1)
计量
- 文章访问数: 3586
- HTML全文浏览量: 374
- PDF下载量: 3789
- 被引次数: 0
