Spatio-temporal Co-variant Hybrid Deep Learning Framework for Cognitive Performance Prediction
-
摘要: 认知表现预测已经成为当前大脑研究的重要课题. 功能磁共振成像技术由于同时具有较好的时间和空间分辨率, 有潜力为认知表现预测提供数据支持. 为了解决基于功能磁共振成像数据对认知表现进行预测时大脑所具有的时−空共变难刻画问题, 提出了一种新型基于大脑学习机制的时−空共变混合深度学习模型, 即深度稀疏自编码器与循环全连接网络混合模型, 以混合神经网络模型的损失函数误差作为认知表现预测能力的评价标准. 在人类连接组项目数据集上的实验结果表明, 提出的时−空共变混合模型能够有效和稳健地预测认知表现, 并提取到与人脑学习、记忆相关的有意义的脑影像特征, 从而为认知表现预测提供技术支持.
-
关键词:
- 循环自编码器 /
- 时−空共变深度学习模型 /
- 混合深度学习模型 /
- 认知表现预测 /
- 脑启发模型
Abstract: Cognitive performance prediction has been an important topic for brain research. Functional magnetic resonance imaging is with high resolution in both spatial and temporal dimensions, which has the potential to support cognitive performance prediction. In order to address the problem that it is hard to characterize the spatio-temporal co-variation of brain data when predicting cognitive performance with functional magnetic resonance imaging data, inspired by the brain learning mechanism, a novel spatio-temporal co-variant hybrid deep learning framework has been presented here for evaluation the cognitive performance prediction, named as deep sparse recurrent autoencoder-recurrent fully connected net, to jointly minimize the loss function of the hybrid neural network models. The experimental results on the Human Connectome Project data set have shown that our proposed framework can predict cognitive performance and learn brain studying and memory-related neuroimaging features effectively and robustly, which can support predicting cognitive performance effectively. -
图 1 基于大脑学习机制的时−空共变混合DSRAE-RFNet模型框架
Fig. 1 The overview of learning mechanism based spatio-temporal co-variant hybrid deep learning framework (DSRAE-RFNet)
图 2 DSRAE-RFNet模型在3组被试上对RT、ACC分别进行预测时的MSE损失图
Fig. 2 The MSE loss when predicting RT and ACC with DSRAE-RFNet model on three groups participants
图 3 DSRAE-RFNet模型在3组被试上对RT、ACC的预测结果
Fig. 3 The predictive results of RT and ACC with DSRAE-RFNet model on three groups participants
图 5 反应时间与准确率表现预测过程中习得的大脑时−空共变特征
Fig. 5 The brain spatio-temporal co-variant features learned from the RT and ACC performance prediction processes
图 7 DSRAE-RFNet及比较算法在单个GPU上单个运行次数的运行时间
Fig. 7 Running time of DSRAE-RFNet and comparable methods on a single GPU during one epoch
表 1 工作记忆任务: fMRI数据信息
Table 1 Working memory task: fMRI data information
参数 任务信息 时间点数量 405 个 扫描持续时间 5 分 01 秒 任务组块数量 8 个 刺激名称 0-back: 被试判断当前呈现内容是否
与预先规定内容一致2-back: 被试判断当前呈现内容是否
与 2 个位置之前的呈现内容一致cue: 任务初始阶段或 block 间的间隔 
下载: 导出CSV
表 2 工作记忆任务: 认知表现数据信息
Table 2 Working memory task: Cognitive performance data information
刺激 反应时间 (RT) 准确率 (ACC) 0-back 总体反应时间 总体准确率 人体刺激反应时间 人体刺激准确率 面孔刺激反应时间 面孔刺激准确率 地点刺激反应时间 地点刺激准确率 工具刺激反应时间 工具刺激准确率 2-back 总体反应时间 总体准确率 人体刺激反应时间 人体刺激准确率 面孔刺激反应时间 面孔刺激准确率 地点刺激反应时间 地点刺激准确率 工具刺激反应时间 工具刺激准确率
下载: 导出CSV
表 3 与未采用学习机制模型比较的认知表现预测结果
Table 3 Cognitive performance prediction results compared with the model without learning mechanism
认知表现 组别 没有采用学习机制 采用学习机制 RT 第 1 组 0.455 0.700 第 2 组 0.330 0.740 第 3 组 0.680 0.776 ACC 第 1 组 0.449 0.429 第 2 组 0.388 0.477 第 3 组 0.523 0.536
下载: 导出CSV
表 4 与其他预测模型比较的认知表现预测结果
Table 4 Cognitive performance prediction results compared with the other predictive model
组别 RT ACC 第 1 组 第 2 组 第 3 组 第 1 组 第 2 组 第 3 组 ICA 0.677 0.661 0.765 0.090 0.505 0.449 GLM 0.451 0.369 0.292 0.232 0.224 0.343 RNN 0.696 0.747 0.719 0.390 0.453 0.511 LSTM 0.685 0.674 0.731 0.369 0.368 0.345 AE 0.078 0.519 0.663 0.178 0.289 0.202 DSRNN[33] 0.692 0.576 0.733 0.413 0.354 0.356 DVAE[34] 0.157 0.644 0.512 0.411 0.242 0.216 STAAE[35] 0.588 0.431 0.532 0.066 0.122 0.466 DCAE[10] 0.384 0.438 0.465 0.225 0.200 0.089 本文算法 0.700 0.740 0.776 0.429 0.477 0.536
下载: 导出CSV
表 5 对不同认知任务的认知表现预测结果
Table 5 Cognitive performance prediction results for different cognitive tasks
认知表现 组别 情感任务 语言任务 关系任务 工作记忆任务 RT 第1组 0.160 0.720 0.667 0.700 第2组 0.280 0.800 0.990 0.740 第3组 0.240 0.720 0.760 0.776 ACC 第1组 0.400 0.520 0.693 0.429 第2组 0.320 0.720 0.820 0.477 第3组 0.140 0.760 0.840 0.536
下载: 导出CSV
-
[1] Chhatwal J P, Schultz A P, Dang Y, Ostaszewski B, Liu L, Yang H S, et al. Plasma N-terminal tau fragment levels predict future cognitive decline and neurodegeneration in healthy elderly individuals. Nature Communications, 2020, 11(1): 1-10 doi: 10.1038/s41467-019-13993-7 [2] Mayer R E. How can brain research inform academic learning and instruction? Educational Psychology Review, 2017, 29(4): 835-846 doi: 10.1007/s10648-016-9391-1 [3] Sigman M, Pena M, Goldin A P, Ribeiro S. Neuroscience and education: Prime time to build the bridge. Nature Neuroscience, 2014, 17(4): 497-502 doi: 10.1038/nn.3672 [4] Cao P, Liu X, Yang J, Zhao D, Huang M, Zaiane O. $ {\cal{L}}$ 2, 1-$ {\cal{L}}$ 1 regularized nonlinear multi-task representation learning based cognitive performance prediction of Alzheimer’s disease. Pattern Recognition, 2018, 17: 195-215[5] Jiang X, Zhao L, Liu H, Guo L, Kendrick K M, Liu T. A cortical folding pattern-guided model of intrinsic functional brain networks in emotion processing. Frontiers in Neuroscience, 2018, 12: 575 doi: 10.3389/fnins.2018.00575 [6] Feng C, Yuan J, Geng H, Gu R, Zhou H, Wu X, et al. Individualized prediction of trait narcissism from whole-brain resting-state functional connectivity. Human Brain Mapping, 2018, 39(9): 3701-3712 doi: 10.1002/hbm.24205 [7] Dargan S, Kumar M, Ayyagari M R, Kumar G. A survey of deep learning and its applications: A new paradigm to machine learning. Archives of Computational Methods in Engineering, 2019: 1-22 [8] 刘小波, 刘鹏, 蔡之华, 乔禹霖, 王凌, 汪敏. 基于深度学习的光学遥感图像目标检测研究进展. 自动化学报, 2021, 47(9): 2078-2089 doi: 10.16383/j.aas.c190455Liu Xiao-Bo, Liu Peng, Cai Zhi-Hua, Qiao Yu-Lin, Wang Ling, Wang Min. Research progress of optical remote sensing image object detection based on deep learning. Acta Automatica Sinica, 2021, 47(9): 2078-2089 doi: 10.16383/j.aas.c190455 [9] Hu X, Huang H, Peng B, Han J, Liu N, Lv J, et al. Latent source mining in fMRI via restricted boltzmann machine. Human Brain Mapping, 2018, 39(6): 2368-2380 doi: 10.1002/hbm.24005 [10] Huang H, Hu X, Zhao Y, Makkie M, Dong Q, Zhao S, et al. Modeling task fMRI data via deep convolutional autoencoder. IEEE Transactions on Medical Imaging, 2018, 37(7): 1551-1561 doi: 10.1109/TMI.2017.2715285 [11] Li Q, Dong Q, Ge F, Qiang N, Zhao Y, Wang H, et al. Simultaneous spatial-temporal decomposition of connectome-scale brain networks by deep sparse recurrent auto-encoders. In: Proceedings of the 2019 International Conference on Information Processing in Medical Imaging. Hongkong, China: 2019. 579−591 [12] Viejo G, Khamassi M, Brovelli A, Girard B. Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Frontiers in Behavioral Neuroscience, 2015, 9: 225 [13] 顾凌云, 吕文志, 杨勇, 高军峰, 官金安, 周到. 基于PCANet和SVM的谎言测试研究. 电子学报, 2016, 44(8): 1969-1973 doi: 10.3969/j.issn.0372-2112.2016.08.028Gu Ling-Yun, Lv Wen-Zhi, Yang Yong, Gao Jun-Feng, Guan Jin-An, Zhou Dao. Deception detection study based on PCANet and support vector machine. Acta Elctronica Sinica, 2016, 44(8): 1969-1973 doi: 10.3969/j.issn.0372-2112.2016.08.028 [14] Betzel R F, Bassett D S. Multi-scale brain networks. NeuroImage, 2017, 160: 73-83 doi: 10.1016/j.neuroimage.2016.11.006 [15] Huo Z, Shen D, Huang H. New multi-task learning model to predict Alzheimer＇s disease cognitive assessment. In: Proceedings of the 2016 Medical Image Computing and Computer Assisted Int-ervention. Lima, Peru: 2016. 317−25 [16] Hassabis D, Kumaran D, Summerfield C, Botvinick M. Neuroscience-inspired artificial intelligence. Neuron, 2017, 95(2): 245-258 doi: 10.1016/j.neuron.2017.06.011 [17] Li Q, Dong Q, Ge F, Qiang N, Wu X, Liu T. Simultaneous Spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder. Brain Imaging and Behavior, 2021, 15: 2646-2660 doi: 10.1007/s11682-021-00469-w [18] Barch D M, Burgess G C, Harms M P, Petersen S E, Schlaggar B L, Corbetta M, et al. Function in the human connectome: Task-fMRI and individual differences in behavior. NeuroImage, 2013, 80: 169-189 doi: 10.1016/j.neuroimage.2013.05.033 [19] Glasser M F, Sotiropoulos S N, Wilson J A, Coalson T S, Fischl B, Andersson J L, et al. The minimal preprocessing pipelines for the human connectome project. NeuroImage, 2013, 80: 105-124 doi: 10.1016/j.neuroimage.2013.04.127 [20] Drobyshevsky A, Baumann S B, Schneider W. A rapid fMRI task battery for mapping of visual, motor, cognitive, and emotional function. NeuroImage, 2006, 31: 732-744 doi: 10.1016/j.neuroimage.2005.12.016 [21] Caceres A, Hall D L, Zelaya F O, Williams S C R, Mehta M A. Measuring fMRI reliability with the intra-class correlation coefficient. NeuroImage, 2009, 45(3): 758-768 doi: 10.1016/j.neuroimage.2008.12.035 [22] Colan S D. The why and how of Z scores. Journal of the American Society of Ehocardiography, 2013, 26(1): 38-40 doi: 10.1016/j.echo.2012.11.005 [23] Kingma D P, Ba J L. ADAM: A method for stochastic optimization. In: Proceedings of the 2015 International Conference on Learning Representations. San Diego, USA. 2015. 1−15 [24] Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B, 2005, 67(2): 301-320 doi: 10.1111/j.1467-9868.2005.00503.x [25] Paz-Linares D, Vega-Hernández M, Rojas-López P A, Valdés-Hernández P A, Martínez-Montes E, Valdés-Sosa P A. Spatio temporal EEG source imaging with the hierarchical Bayesian elastic net and elitist lasso models. Frontiers in Neuroscience, 2017, 11: 1-22 [26] Spisak T, Kincses B, Schlitt F, Zunhammer M, Schmidt-Wilcke T, Kincses Z T, et al. Pain-free resting-state functional brain connectivity predicts individual pain sensitivity. Nature Communications, 2020, 11: 187 doi: 10.1038/s41467-019-13785-z [27] Bransford J D, Ann L, Brown A, Cocking R R. How people learn brain, mind, experience, and school committee. Psychology, 2004, 116 [28] Grossberg S. Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world. Neural Networks, 2013, 37: 1-47 doi: 10.1016/j.neunet.2012.09.017 [29] Vaadia E. Lrarning how the brain learns. Nature, 2000, 405: 523-525 [30] Yaple Z A, Stevens W D, Arsalidou M. NeuroImage Meta-analyses of the n-back working memory task? FMRI evidence of age-related changes in prefrontal cortex involvement across the adult lifespan. NeuroImage, 2019, 196: 16-31 doi: 10.1016/j.neuroimage.2019.03.074 [31] Qiang N, Dong Q, Zhang W, Ge B, Ge F, Liang H, et al. Modeling task-based fMRI data via deep belief network with neural architecture search. Computerized Medical Imaging and Graphics, 2020: 101747 [32] Wen X, Wang H, Liu Z, Liu C, Li K, Ding M, et al. Dynamic top-down configuration by the core control system during working memory. Neuroscience, 2018, 391: 13-24 doi: 10.1016/j.neuroscience.2018.09.004 [33] Wang H, Zhao S, Dong Q, Cui Y, Chen Y, Han J, et al. Recognizing brain states using deep sparse recurrent neural network. IEEE Transactions on Medical Imaging, 2018, 38(4): 1058-1068 [34] Qiang N, Dong Q, Ge F, Liang H, Ge B, Zhang S, et al. Deep variational autoencoder for mapping functional brain networks. IEEE Transactions on Cognitive and Developmental Systems, 2021, 13(4): 841-852 doi: 10.1109/TCDS.2020.3025137 [35] Dong Q, Qiang N, Lv J, Li X, Liu T, Li Q. Spatiotemporal attention autoencoder (STAAE) for ADHD classification. In: Proceedings of the 2020 Medical Image Computing and Computer-Assisted Intervention. Lima, Peru: 2020. 508−517 [36] Li Q, Wu X, Xu L, Chen K, Yao L. Classification of Alzheimer’s disease, mild cognitive impairment, and cognitively unimpaired individuals using multi-feature kernel discriminant dictionary learning. Frontiers in Computational Neuroscience, 2018, 11: 117 doi: 10.3389/fncom.2017.00117 [37] Wu X, Li Q, Xu L, Chen K, Yao L. Multi-feature kernel discriminant dictionary learning for face recognition. Pattern Recognition, 2017, 66: 404-411 doi: 10.1016/j.patcog.2016.12.001 [38] 张宪法, 郝矿荣, 陈磊. 免疫多域特征融合的多核学习SVM运动想象脑电信号分类. 自动化学报, 2020, 46(11): 2417-2426 doi: 10.16383/j.aas.c180247Zhang Xian-Fa, Hao Kuang-Rong, Chen Lei. Motor imagery EEG classification based on immune multi-domain-feature fusion and multiple kernel learning SVM. Acta Automatica Sinica, 2020, 46(11): 2417-2426 doi: 10.16383/j.aas.c180247 [39] Li Q, Zhang W, Zhao L, Wu X, Liu T. Evolutional neural architecture search for optimization of spatiotemporal brain network decomposition. IEEE Transactions on Biomedical Engineering, 2022, 69(2): 624-634 doi: 10.1109/TBME.2021.3102466 [40] 冀俊忠, 邹爱笑, 刘金铎. 基于功能磁共振成像的人脑效应连接网络识别方法综述. 自动化学报, 2021, 47(2): 278-296 doi: 10.16383/j.aas.c190491Ji Jun-Zhong, Zou Ai-Xiao, Liu Jin-Duo. An overview of identification methods on human brain effective connectivity networks based on functional magnetic resonance imaging. Acta Automatica Sinica, 2021, 47(2): 278-296 doi: 10.16383/j.aas.c190491 -
图(7) / 表(5)
计量
- 文章访问数: 677
- HTML全文浏览量: 189
- PDF下载量: 183
- 被引次数: 0
