2.845

2023影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

面向认知表现预测的时−空共变混合深度学习模型

李晴 徐雪远 邬霞

李晴, 徐雪远, 邬霞. 面向认知表现预测的时−空共变混合深度学习模型. 自动化学报, 2022, 48(12): 2931−2940 doi: 10.16383/j.aas.c220025
引用本文: 李晴, 徐雪远, 邬霞. 面向认知表现预测的时−空共变混合深度学习模型. 自动化学报, 2022, 48(12): 2931−2940 doi: 10.16383/j.aas.c220025
Li Qing, Xu Xue-Yuan, Wu Xia. Spatio-temporal co-variant hybrid deep learning framework for cognitive performance prediction. Acta Automatica Sinica, 2022, 48(12): 2931−2940 doi: 10.16383/j.aas.c220025
Citation: Li Qing, Xu Xue-Yuan, Wu Xia. Spatio-temporal co-variant hybrid deep learning framework for cognitive performance prediction. Acta Automatica Sinica, 2022, 48(12): 2931−2940 doi: 10.16383/j.aas.c220025

面向认知表现预测的时−空共变混合深度学习模型

doi: 10.16383/j.aas.c220025
基金项目: 北京市自然科学基金(4212037)资助
详细信息
    作者简介:

    李晴:北京师范大学认知神经科学与学习国家重点实验室博士后. 2022年获北京师范大学博士学位. 主要研究方向为脑影像智能分析. E-mail: liqing@bnu.edu.cn

    徐雪远:2022年获北京师范大学人工智能学院博士学位. 主要研究方向为脑信号智能分析. E-mail: xuxueyuan@mail.bnu.edu.cn

    邬霞:北京师范大学人工智能学院教授. 2008年获北京师范大学认知神经科学与学习研究所博士学位. 主要研究方向为医学图像处理. 本文通信作者. E-mail: wuxia@bnu.edu.cn

Spatio-temporal Co-variant Hybrid Deep Learning Framework for Cognitive Performance Prediction

Funds: Supported by Natural Science Foundation of Beijing (4212037)
More Information
    Author Bio:

    LI Qing Postdoctoral researcher at the State Key Laboratory of Co-gnitive Neuroscience and Learning, Beijing Normal University. She received her Ph.D. degree from Beijing Normal University in 2022. Her main research interest is brain imaging intelligent analysis

    XU Xue-Yuan He received his Ph.D. degree from Beijing Normal University in 2022. His main resea-rch interest is brain signal intelligent analysis

    WU Xia Professor at the School of Artificial Intelligence, Beijing Normal University. She received her Ph.D. degree from Beijing Normal University in 2008. Her main research interest is medical image processing. Corresponding author of this paper

  • 摘要: 认知表现预测已经成为当前大脑研究的重要课题. 功能磁共振成像技术由于同时具有较好的时间和空间分辨率, 有潜力为认知表现预测提供数据支持. 为了解决基于功能磁共振成像数据对认知表现进行预测时大脑所具有的时−空共变难刻画问题, 提出了一种新型基于大脑学习机制的时−空共变混合深度学习模型, 即深度稀疏自编码器与循环全连接网络混合模型, 以混合神经网络模型的损失函数误差作为认知表现预测能力的评价标准. 在人类连接组项目数据集上的实验结果表明, 提出的时−空共变混合模型能够有效和稳健地预测认知表现, 并提取到与人脑学习、记忆相关的有意义的脑影像特征, 从而为认知表现预测提供技术支持.
  • 图  1  基于大脑学习机制的时−空共变混合DSRAE-RFNet模型框架

    Fig.  1  The overview of learning mechanism based spatio-temporal co-variant hybrid deep learning framework (DSRAE-RFNet)

    图  2  DSRAE-RFNet模型在3组被试上对RT、ACC分别进行预测时的MSE损失图

    Fig.  2  The MSE loss when predicting RT and ACC with DSRAE-RFNet model on three groups participants

    图  3  DSRAE-RFNet模型在3组被试上对RT、ACC的预测结果

    Fig.  3  The predictive results of RT and ACC with DSRAE-RFNet model on three groups participants

    图  4  反应时间表现预测结果

    Fig.  4  Reaction time performance predicting results

    图  5  反应时间与准确率表现预测过程中习得的大脑时−空共变特征

    Fig.  5  The brain spatio-temporal co-variant features learned from the RT and ACC performance prediction processes

    图  6  准确率表现预测结果

    Fig.  6  Accuracy performance predicting results

    图  7  DSRAE-RFNet及比较算法在单个GPU上单个运行次数的运行时间

    Fig.  7  Running time of DSRAE-RFNet and comparable methods on a single GPU during one epoch

    表  1  工作记忆任务: fMRI数据信息

    Table  1  Working memory task: fMRI data information

    参数任务信息
    时间点数量405 个
    扫描持续时间5 分 01 秒
    任务组块数量8 个
    刺激名称0-back: 被试判断当前呈现内容是否
    与预先规定内容一致
    2-back: 被试判断当前呈现内容是否
    与 2 个位置之前的呈现内容一致
    cue: 任务初始阶段或 block 间的间隔
    下载: 导出CSV

    表  2  工作记忆任务: 认知表现数据信息

    Table  2  Working memory task: Cognitive performance data information

    刺激反应时间 (RT)准确率 (ACC)
    0-back总体反应时间 总体准确率
    人体刺激反应时间 人体刺激准确率
    面孔刺激反应时间 面孔刺激准确率
    地点刺激反应时间 地点刺激准确率
    工具刺激反应时间 工具刺激准确率
    2-back总体反应时间 总体准确率
    人体刺激反应时间 人体刺激准确率
    面孔刺激反应时间 面孔刺激准确率
    地点刺激反应时间 地点刺激准确率
    工具刺激反应时间 工具刺激准确率
    下载: 导出CSV

    表  3  与未采用学习机制模型比较的认知表现预测结果

    Table  3  Cognitive performance prediction results compared with the model without learning mechanism

    认知表现组别没有采用学习机制采用学习机制
    RT第 1 组0.4550.700
    第 2 组0.3300.740
    第 3 组0.6800.776
    ACC第 1 组0.4490.429
    第 2 组0.3880.477
    第 3 组0.5230.536
    下载: 导出CSV

    表  4  与其他预测模型比较的认知表现预测结果

    Table  4  Cognitive performance prediction results compared with the other predictive model

    组别RTACC
    第 1 组第 2 组第 3 组第 1 组第 2 组第 3 组
    ICA0.6770.6610.7650.0900.5050.449
    GLM0.4510.3690.2920.2320.2240.343
    RNN0.6960.7470.7190.3900.4530.511
    LSTM0.6850.6740.7310.3690.3680.345
    AE0.0780.5190.6630.1780.2890.202
    DSRNN[33]0.6920.5760.7330.4130.3540.356
    DVAE[34]0.1570.6440.5120.4110.2420.216
    STAAE[35]0.5880.4310.5320.0660.1220.466
    DCAE[10]0.3840.4380.4650.2250.2000.089
    本文算法0.7000.7400.7760.4290.4770.536
    下载: 导出CSV

    表  5  对不同认知任务的认知表现预测结果

    Table  5  Cognitive performance prediction results for different cognitive tasks

    认知表现组别情感任务语言任务关系任务工作记忆任务
    RT第1组0.1600.7200.6670.700
    第2组0.2800.8000.9900.740
    第3组0.2400.7200.7600.776
    ACC第1组0.4000.5200.6930.429
    第2组0.3200.7200.8200.477
    第3组0.1400.7600.8400.536
    下载: 导出CSV
  • [1] Chhatwal J P, Schultz A P, Dang Y, Ostaszewski B, Liu L, Yang H S, et al. Plasma N-terminal tau fragment levels predict future cognitive decline and neurodegeneration in healthy elderly individuals. Nature Communications, 2020, 11(1): 1-10 doi: 10.1038/s41467-019-13993-7
    [2] Mayer R E. How can brain research inform academic learning and instruction? Educational Psychology Review, 2017, 29(4): 835-846 doi: 10.1007/s10648-016-9391-1
    [3] Sigman M, Pena M, Goldin A P, Ribeiro S. Neuroscience and education: Prime time to build the bridge. Nature Neuroscience, 2014, 17(4): 497-502 doi: 10.1038/nn.3672
    [4] Cao P, Liu X, Yang J, Zhao D, Huang M, Zaiane O. $ {\cal{L}}$2, 1-$ {\cal{L}}$1 regularized nonlinear multi-task representation learning based cognitive performance prediction of Alzheimer’s disease. Pattern Recognition, 2018, 17: 195-215
    [5] Jiang X, Zhao L, Liu H, Guo L, Kendrick K M, Liu T. A cortical folding pattern-guided model of intrinsic functional brain networks in emotion processing. Frontiers in Neuroscience, 2018, 12: 575 doi: 10.3389/fnins.2018.00575
    [6] Feng C, Yuan J, Geng H, Gu R, Zhou H, Wu X, et al. Individualized prediction of trait narcissism from whole-brain resting-state functional connectivity. Human Brain Mapping, 2018, 39(9): 3701-3712 doi: 10.1002/hbm.24205
    [7] Dargan S, Kumar M, Ayyagari M R, Kumar G. A survey of deep learning and its applications: A new paradigm to machine learning. Archives of Computational Methods in Engineering, 2019: 1-22
    [8] 刘小波, 刘鹏, 蔡之华, 乔禹霖, 王凌, 汪敏. 基于深度学习的光学遥感图像目标检测研究进展. 自动化学报, 2021, 47(9): 2078-2089 doi: 10.16383/j.aas.c190455

    Liu Xiao-Bo, Liu Peng, Cai Zhi-Hua, Qiao Yu-Lin, Wang Ling, Wang Min. Research progress of optical remote sensing image object detection based on deep learning. Acta Automatica Sinica, 2021, 47(9): 2078-2089 doi: 10.16383/j.aas.c190455
    [9] Hu X, Huang H, Peng B, Han J, Liu N, Lv J, et al. Latent source mining in fMRI via restricted boltzmann machine. Human Brain Mapping, 2018, 39(6): 2368-2380 doi: 10.1002/hbm.24005
    [10] Huang H, Hu X, Zhao Y, Makkie M, Dong Q, Zhao S, et al. Modeling task fMRI data via deep convolutional autoencoder. IEEE Transactions on Medical Imaging, 2018, 37(7): 1551-1561 doi: 10.1109/TMI.2017.2715285
    [11] Li Q, Dong Q, Ge F, Qiang N, Zhao Y, Wang H, et al. Simultaneous spatial-temporal decomposition of connectome-scale brain networks by deep sparse recurrent auto-encoders. In: Proceedings of the 2019 International Conference on Information Processing in Medical Imaging. Hongkong, China: 2019. 579−591
    [12] Viejo G, Khamassi M, Brovelli A, Girard B. Modeling choice and reaction time during arbitrary visuomotor learning through the coordination of adaptive working memory and reinforcement learning. Frontiers in Behavioral Neuroscience, 2015, 9: 225
    [13] 顾凌云, 吕文志, 杨勇, 高军峰, 官金安, 周到. 基于PCANet和SVM的谎言测试研究. 电子学报, 2016, 44(8): 1969-1973 doi: 10.3969/j.issn.0372-2112.2016.08.028

    Gu Ling-Yun, Lv Wen-Zhi, Yang Yong, Gao Jun-Feng, Guan Jin-An, Zhou Dao. Deception detection study based on PCANet and support vector machine. Acta Elctronica Sinica, 2016, 44(8): 1969-1973 doi: 10.3969/j.issn.0372-2112.2016.08.028
    [14] Betzel R F, Bassett D S. Multi-scale brain networks. NeuroImage, 2017, 160: 73-83 doi: 10.1016/j.neuroimage.2016.11.006
    [15] Huo Z, Shen D, Huang H. New multi-task learning model to predict Alzheimer's disease cognitive assessment. In: Proceedings of the 2016 Medical Image Computing and Computer Assisted Int-ervention. Lima, Peru: 2016. 317−25
    [16] Hassabis D, Kumaran D, Summerfield C, Botvinick M. Neuroscience-inspired artificial intelligence. Neuron, 2017, 95(2): 245-258 doi: 10.1016/j.neuron.2017.06.011
    [17] Li Q, Dong Q, Ge F, Qiang N, Wu X, Liu T. Simultaneous Spatial-temporal decomposition for connectome-scale brain networks by deep sparse recurrent auto-encoder. Brain Imaging and Behavior, 2021, 15: 2646-2660 doi: 10.1007/s11682-021-00469-w
    [18] Barch D M, Burgess G C, Harms M P, Petersen S E, Schlaggar B L, Corbetta M, et al. Function in the human connectome: Task-fMRI and individual differences in behavior. NeuroImage, 2013, 80: 169-189 doi: 10.1016/j.neuroimage.2013.05.033
    [19] Glasser M F, Sotiropoulos S N, Wilson J A, Coalson T S, Fischl B, Andersson J L, et al. The minimal preprocessing pipelines for the human connectome project. NeuroImage, 2013, 80: 105-124 doi: 10.1016/j.neuroimage.2013.04.127
    [20] Drobyshevsky A, Baumann S B, Schneider W. A rapid fMRI task battery for mapping of visual, motor, cognitive, and emotional function. NeuroImage, 2006, 31: 732-744 doi: 10.1016/j.neuroimage.2005.12.016
    [21] Caceres A, Hall D L, Zelaya F O, Williams S C R, Mehta M A. Measuring fMRI reliability with the intra-class correlation coefficient. NeuroImage, 2009, 45(3): 758-768 doi: 10.1016/j.neuroimage.2008.12.035
    [22] Colan S D. The why and how of Z scores. Journal of the American Society of Ehocardiography, 2013, 26(1): 38-40 doi: 10.1016/j.echo.2012.11.005
    [23] Kingma D P, Ba J L. ADAM: A method for stochastic optimization. In: Proceedings of the 2015 International Conference on Learning Representations. San Diego, USA. 2015. 1−15
    [24] Zou H, Hastie T. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society Series B, 2005, 67(2): 301-320 doi: 10.1111/j.1467-9868.2005.00503.x
    [25] Paz-Linares D, Vega-Hernández M, Rojas-López P A, Valdés-Hernández P A, Martínez-Montes E, Valdés-Sosa P A. Spatio temporal EEG source imaging with the hierarchical Bayesian elastic net and elitist lasso models. Frontiers in Neuroscience, 2017, 11: 1-22
    [26] Spisak T, Kincses B, Schlitt F, Zunhammer M, Schmidt-Wilcke T, Kincses Z T, et al. Pain-free resting-state functional brain connectivity predicts individual pain sensitivity. Nature Communications, 2020, 11: 187 doi: 10.1038/s41467-019-13785-z
    [27] Bransford J D, Ann L, Brown A, Cocking R R. How people learn brain, mind, experience, and school committee. Psychology, 2004, 116
    [28] Grossberg S. Adaptive resonance theory: How a brain learns to consciously attend, learn, and recognize a changing world. Neural Networks, 2013, 37: 1-47 doi: 10.1016/j.neunet.2012.09.017
    [29] Vaadia E. Lrarning how the brain learns. Nature, 2000, 405: 523-525
    [30] Yaple Z A, Stevens W D, Arsalidou M. NeuroImage Meta-analyses of the n-back working memory task? FMRI evidence of age-related changes in prefrontal cortex involvement across the adult lifespan. NeuroImage, 2019, 196: 16-31 doi: 10.1016/j.neuroimage.2019.03.074
    [31] Qiang N, Dong Q, Zhang W, Ge B, Ge F, Liang H, et al. Modeling task-based fMRI data via deep belief network with neural architecture search. Computerized Medical Imaging and Graphics, 2020: 101747
    [32] Wen X, Wang H, Liu Z, Liu C, Li K, Ding M, et al. Dynamic top-down configuration by the core control system during working memory. Neuroscience, 2018, 391: 13-24 doi: 10.1016/j.neuroscience.2018.09.004
    [33] Wang H, Zhao S, Dong Q, Cui Y, Chen Y, Han J, et al. Recognizing brain states using deep sparse recurrent neural network. IEEE Transactions on Medical Imaging, 2018, 38(4): 1058-1068
    [34] Qiang N, Dong Q, Ge F, Liang H, Ge B, Zhang S, et al. Deep variational autoencoder for mapping functional brain networks. IEEE Transactions on Cognitive and Developmental Systems, 2021, 13(4): 841-852 doi: 10.1109/TCDS.2020.3025137
    [35] Dong Q, Qiang N, Lv J, Li X, Liu T, Li Q. Spatiotemporal attention autoencoder (STAAE) for ADHD classification. In: Proceedings of the 2020 Medical Image Computing and Computer-Assisted Intervention. Lima, Peru: 2020. 508−517
    [36] Li Q, Wu X, Xu L, Chen K, Yao L. Classification of Alzheimer’s disease, mild cognitive impairment, and cognitively unimpaired individuals using multi-feature kernel discriminant dictionary learning. Frontiers in Computational Neuroscience, 2018, 11: 117 doi: 10.3389/fncom.2017.00117
    [37] Wu X, Li Q, Xu L, Chen K, Yao L. Multi-feature kernel discriminant dictionary learning for face recognition. Pattern Recognition, 2017, 66: 404-411 doi: 10.1016/j.patcog.2016.12.001
    [38] 张宪法, 郝矿荣, 陈磊. 免疫多域特征融合的多核学习SVM运动想象脑电信号分类. 自动化学报, 2020, 46(11): 2417-2426 doi: 10.16383/j.aas.c180247

    Zhang Xian-Fa, Hao Kuang-Rong, Chen Lei. Motor imagery EEG classification based on immune multi-domain-feature fusion and multiple kernel learning SVM. Acta Automatica Sinica, 2020, 46(11): 2417-2426 doi: 10.16383/j.aas.c180247
    [39] Li Q, Zhang W, Zhao L, Wu X, Liu T. Evolutional neural architecture search for optimization of spatiotemporal brain network decomposition. IEEE Transactions on Biomedical Engineering, 2022, 69(2): 624-634 doi: 10.1109/TBME.2021.3102466
    [40] 冀俊忠, 邹爱笑, 刘金铎. 基于功能磁共振成像的人脑效应连接网络识别方法综述. 自动化学报, 2021, 47(2): 278-296 doi: 10.16383/j.aas.c190491

    Ji Jun-Zhong, Zou Ai-Xiao, Liu Jin-Duo. An overview of identification methods on human brain effective connectivity networks based on functional magnetic resonance imaging. Acta Automatica Sinica, 2021, 47(2): 278-296 doi: 10.16383/j.aas.c190491
  • 加载中
图(7) / 表(5)
计量
  • 文章访问数:  662
  • HTML全文浏览量:  184
  • PDF下载量:  181
  • 被引次数: 0
出版历程
  • 收稿日期:  2022-01-10
  • 录用日期:  2022-06-16
  • 网络出版日期:  2022-09-09
  • 刊出日期:  2022-12-23

目录

    /

    返回文章
    返回