前额叶皮层启发的Transformer模型应用及其进展

潘雨辰; 贾克斌; 张铁林

doi:10.16383/j.aas.c240538

前额叶皮层启发的Transformer模型应用及其进展

doi: 10.16383/j.aas.c240538 cstr: 32138.14.j.aas.c240538

潘雨辰^{1, 2,},
贾克斌^2,,
张铁林^{1, 3, 4,}

1.
中国科学院脑科学与智能技术卓越创新中心上海 200031
2.
北京工业大学北京 100124
3.
中国科学院大学北京 100049
4.
中国科学院自动化研究所北京100190

基金项目: 北京市科技新星(20230484369), 上海市市级科技重大专项(2021SHZDZX), 中科院青促会, 多模态人工智能系统全国重点实验室开放课题基金等资助.

详细信息

作者简介:
潘雨辰：北京工业大学信息科学技术学院硕士研究生, 中科院脑科学与智能技术卓越创新中心联合培养学生. 2019年获得北京工业大学工学学士学位. 主要研究方向为类脑模型算法.E-mail: 18201335023@sina.cn

贾克斌：北京工业大学信息科学技术学院教授, 博士. 主要研究方向为图像/视频处理技术与生物医学信息处理技术.E-mail: kebinj@bjut.edu.cn

张铁林：中国科学院脑智卓越中心, 脑认知与类脑智能国重实验室研究员, 课题组长, 兼职中科院自动化所复杂系统认知与决策实验室. 主要从事类脑脉冲神经网络算法, 类脑芯片及AI for Neuroscience研究. 本文通信作者.E-mail: zhangtielin@ion.ac.cn

计量
- 文章访问数: 172
- HTML全文浏览量: 77
- 被引次数: 0
出版历程
- 收稿日期: 2024-12-13
- 录用日期: 2024-12-13
- 网络出版日期: 2025-03-03

The Application and Progress of Prefrontal Cortex-inspired Transformer Model

PAN Yu-Chen^{1, 2
,},
JIA Ke-Bin^2
,,
ZHANG Tie-lin^{1, 3, 4
,}

1.
Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences, Shanghai 200031
2.
Beijing University of Technology, Beijing 100124
3.
University of Chinese Academy of Sciences, Beijing 100049
4.
Institute of Automation, Chinese Academy of Sciences, Beijing 100190

Funds: Supported by Beijing Nova Program (20230484369), Shanghai Municipal Science and Technology Major Project (2021SHZDZX), Youth Innovation Promotion Association of Chinese Academy of Sciences, and Open Projects Program of State Key Laboratory of Multimodal Artificial Intelligence Systems.

More Information

Author Bio:
PAN Yu-Chen　Master’s student in the School of Information Science and Technology, Beijing University of Technology, co-supervised by the Center for Excellence of Brain Science and Intelligence Technology, Chinese Academy of Sciences. He received his bachelor degree in engineering from Beijing University of Technology in 2019. His research interest covers brain-inspired algorithms

JIA Ke-Bin　Professor at the School of Information Science and Technology, Beijing University of Technology. His research interests are focused on Image/Video coding and processing, bioinformation processing

ZHANG Tie-Lin　Principal indicator at the Key Laboratory of Brain Cognition and Brain-inspired Intelligence, Center for Excellence in Brain Science and Intelligence Technology, Chinese Academy of Sciences (CAS), also a co-PI at Key Laboratory of Complex System for Recognition and Decision-making, Institute of Automation, CAS. He mainly engaged in the research of brain-inspired spiking neural network algorithms, brain-inspired chips, and AI for Neuroscience. Corresponding author of this paper

摘要

摘要: 本文聚焦于生物结构与类脑智能的交叉研究方向, 探讨前额叶皮层的结构及其认知功能对人工智能领域Transformer模型的启发. 前额叶皮层在认知控制和决策制定中扮演着关键角色, 本文首先介绍前额叶皮层的注意力机制、生物编码、多感觉融合等相关生物研究进展, 然后探讨这些生物机制如何启发新型的类脑Transformer架构, 重点提升其在自注意力、位置编码、多模态整合等方面的生物合理性与计算高效性. 最后, 总结前额叶皮层启发的类脑新模型, 在支持多类型神经网络组合、多领域应用、世界模型构建等方面的发展与潜力, 为生物和人工智能两大领域之间交叉融合构建桥梁.
- 生物结构 /
- 类脑智能 /
- 前额叶皮层 /
- Transformer /
- 世界模型
Abstract: This article focuses on the integration of biological network connectomes and brain-inspired intelligence, exploring how the structure and cognitive functions of the prefrontal cortex can inspire transformer models in the field of artificial intelligence. The prefrontal cortex plays a critical role in cognitive sensory, memory and decision-making. Firstly, the article introduces some advancements of biological research related to the prefrontal cortex’s attention mechanisms, biological encoding, and multisensory integration. Then, it discusses how these biological mechanisms can inspire novel brain-inspired transformer architectures, with a focus on enhancing their biological plausibility and computational efficiency in self-attention, positional encoding, and multimodal integration. Finally, it summarizes the development and potential of new brain-inspired models influenced by the prefrontal cortex, highlighting their support for the integration of various neural network types, multi-domain applications, and the construction of world models in reinforcement learning, thereby building a bridge between the fields of biology and artificial intelligence.
- Biological structures /
- brain-inspired intelligence /
- prefrontal cortex /
- Transformer /
- world model

HTML全文

近年来随着脑机交互、元宇宙概念热度的持续增加, 作为主要的脑机交互技术, 脑电技术备受关注. 脑电采集设备主要分为侵入式和非侵入式两类. 侵入式设备通过手术将微电极阵列植入大脑内部, 能够提供高精度、高信噪比的脑电信号, 但也伴随着手术风险、脑区限制、感染风险和高成本等问题^[1]. 相比之下, 非侵入式设备则通过将电极放置在头皮表面, 避免了手术带来的各种风险和不适, 因此在临床和科研中使用更多.

临床上相比于磁共振和功能近红外等设备, 非侵入式脑电采集设备具有操作简便、安全性高、成本较低、信号实时等优势, 不需要复杂的医疗操作经验就可以实现脑电信号的采集, 显著提高了用户的接受度和使用便利性. 国外脑电行业发展较早, 大部分厂商的非侵入脑电设备体积、重量及操作流程都无法照顾便携可穿戴, 尽管近年来本领域陆续有便携非侵入脑电设备推出, 但如何让受试者更加方便、快速佩戴设备, 同时兼顾脑电信号采集的质量一直是不断优化的方向.

本团队自主研制的非侵入穿戴脑电设备 (以下简称SignBrain设备, 型号P)在对实验人员培训后, 可在3分钟之内完成佩戴及使用 (设备快速佩戴的展示视频链接: https://pan.cstcloud.cn/s/uLpXYelIRbI). SignBrain设备采用爪形结构设计, 前端装配有电极的万向抱紧件 (图5)可根据不同人群头型的大小, 自动调节与头皮的接触角度, 减少佩戴设备调整接触阻抗的时间, 提高实验效率. SignBrain设备采用盐水海绵作为导电介质, 提高使用的便捷性, 佩戴前后无需洗头, 即戴即用. 设备采集的脑电数据通过蓝牙发送至上位机软件, 在线显示脑电波形及实时接触阻抗, 并可随时离线存储或上传至远端服务器进行查看.

SignBrain设备参照JJG1043-2008^[2] (脑电图机检定规程)和GB 9706.226-2021^[3] (医用电气设备第2-26部分: 脑电图机的基本安全和基本性能专用要求)等标准, 已通过国家食品药品监督管理局天津医疗器械质量监督检测中心性能指标检测和安全性检测 (图1), 信号噪声≤1 uV, 共模抑制比为111 dB, 设备能够有效抑制共模噪声和干扰, 提高信号的清晰度和稳定性.

图 1 SignBrain设备性能检测报告

Fig. 1 SignBrain device performance test report

下载: 全尺寸图片幻灯片

设备已在多家医院及相关单位完成小批量的实验测试, 并通过两个脑机交互范式, 高频视觉诱发和想象写字实验验证设备信号的高精度和实时性, 具体实验及数据将在下文中详细介绍.

1. SignBrain穿戴式脑电采集设备

1.1 结构介绍

SignBrain设备结构由主干、爪形、万向抱紧件、电极、海绵等配件构成, 选用合成的改性尼龙材料一体注塑加工而成. 整机重量约为160 g (如图7左图所示的佩戴效果).

1.1.1 主干结构

主干结构 (图2)是双侧爪形及盖板等配件的支撑结构, 提供设备在佩戴中的稳定性. 主干结构的弓形弯折角度, 决定了设备电极与头皮的贴合程度, 经过不断优化主干弯折角度, 进而调整万向抱紧件与头皮接触的压力, 使设备可在长时间佩戴使用时保持良好的贴合度及抗疲劳度. 主干左右侧分别设计有独立的舱室, 左侧舱室用于放置设备电路及射频模块, 右侧为电池舱室.

图 2 设备主干结构

Fig. 2 Device backbone structure

下载: 全尺寸图片幻灯片

1.1.2 爪形结构

SignBrain设备设计有双侧爪形, 共计有两个爪形支体. 单侧爪形设计有8个有效通道, 每一个通道占用爪形结构的一条分支. 总共有16个采集通道、1个参考通道、1个接地通道, 所有通道在爪形结构上设计的位置符合国际10-20标准导联排布^[4] (如图3所示).

图 3 16通道SignBrain设备导联位置排布图

Fig. 3 Layout of the electrodes for 16-channel SignBrain device

下载: 全尺寸图片幻灯片

爪形结构的优势在于佩戴时每一个分支通道可以穿过头发, 不同于脑电帽盖住头发的问题, 即便面对头发较多的女性, 稍微抬起所在分支, 使用棉签拨开头发即可快速降低阻抗, 减少被试在实验前的等待时间.

左侧爪形 (图4(a))包含O1、P7、T7、C3、AF3、F3、F7、FC5、GND, 其中GND电极为偏置电极, 设计在前额叶.

图 4 设备爪形结构

Fig. 4 Claw structure of the device

下载: 全尺寸图片幻灯片

右侧爪形 (图4(b))包含O2、P8、T8、C4、AF4、F4、F8、FC6、REF, 其中REF电极为中心参考电极, 设计在Cz的位置.

SignBrain设备16通道中前额叶脑区包含AF3, AF4, F3, F4, FC5, FC6, F7, F8, 可能涉及情绪调节、注意力和执行功能; 颞叶包含T7, T8可能涉及听觉处理和语言; 枕叶和顶叶有P7、P8、O1、O2, 可能涉及视觉处理.

1.1.3 万向抱紧件

SignBrain设备每一个爪形前端都设计有万向抱紧件, 该部件是设备爪形与头皮接触的重要组件

万向抱紧件 (图5)采用碗状结构, 内部结构为倒锥形, 底部可放置盐水电极及柱形海绵, 盐水电极外侧留有盐水补充口, 及时补充盐水能减少长时间实验带来的基线漂移. 顶部设计有与爪形分支末端锁紧的球形结构, 使万向抱紧件具有30°范围内的万向活动空间.

图 5 万向轴抱紧件

Fig. 5 Universal shaft

下载: 全尺寸图片幻灯片

万向抱紧件内部的盐水电极采用粉末烧结工艺制造^[5]而成, 该电极具有高导电性, 保证信号采集的准确性和稳定性. 粉末烧结材料具有良好的生物相容性, 能减少与皮肤接触时的过敏反应, 适合长期佩戴. 烧结过程可以增强电极的耐腐蚀性, 延长其使用寿命. 设备电极也可以使用常规的Ag/AgCl电极.

万向抱紧件是SignBrain设备在结构上的主要创新之一, 相比传统的脑电帽或爪形结构脑电产品, 电极通过海绵 (或其他导电载体)与头皮的接触角度是固定得, 对于不同大小、形状的头型可能会存在不完全适配. 脑电采集过程中, 影响信号质量的重要因素之一是电极与头皮的接触阻抗, 接触阻抗越大, 引入的噪声越多^[6]. 在本设备中, 由于采用柱形的盐水海绵作为头皮与电极的导电介质, 需与头皮垂直贴合. 因此在爪形结构末端设计有万向组件, 佩戴设备后, 可根据不同头型, 调整抱紧件的角度, 使其与头皮垂直, 保证海绵与头皮接触面积最大, 快速降低接触阻抗, 开展脑电实验.

1.1.4 海绵设计

考虑到不同性别、发量、头型的人群进行实验, 会存在个别通道抱紧件与头皮接触空位情况, 为此设计团队分别设计了短、中、长三种长度海绵 (图6), 设备默认组装短号海绵, 佩戴后若抱紧件与头皮的空位较大 (超过5 mm), 则建议更换中号长度海绵, 当抱紧件与头皮距离超过10 mm, 则建议更换长海绵. 更换海绵无需重新摘下设备, 只需抬起爪形分支, 重新插取海绵即可.

图 6 短中长三种海绵规格展示

Fig. 6 Three types of sponge: short, medium and long

下载: 全尺寸图片幻灯片

1.2 材料介绍

SignBrain设备在实际使用过程中, 需要对设备结构进行频繁的弯折操作 (包括佩戴过程、不同头型的挤压等), 在材料上需要考虑韧性, 没有足够韧性的材料, 主干结构无法夹住头部, 会在实验过程中滑落, 疲劳度较差的材料在几次实验后, 也将失去会弹性, 降低与头部的夹持力度. 我们在结构设计过程以及小批量生产之前进行了大量的材料选型比较, 包括尼龙、PE、ABS等材料, 综合比较发现改性尼龙可以获得好的韧性和回弹效果.

因此, 设备主体材料选用合成的改性尼龙材质, 该材质具有很好的韧性, 能够承受反复弯折而不易断裂或损坏, 大量反复弯折具有良好的抗疲劳性, 能够在长时间使用后保持性能稳定, 并且尼龙具有较好的耐磨损性能, 在一定温度范围内保持其机械性能, 对环境温度变化具有一定的适应性, 适合在摩擦和磨损较大的环境中使用. 尼龙还是一种轻质材料, 非常适合可穿戴^[7].

1.3 硬件设计方案

SignBrain设备可以采集16通道的脑电信号, 支持有线/无线蓝牙方式实时传输脑电数据 (图7), 支持在线阻抗检测及提醒, Marker同步记录, 可连续工作8个小时以上.

图 7 SignBrain设备效果图

Fig. 7 SignBrain device effect picture

下载: 全尺寸图片幻灯片

系统内部设计前级滤波、ADC、低功耗MCU等电路^[8−12]. 信号流程图如图8所示. 前端脑电信号经过前置滤波电路滤波, 由内部集成的两片ADS1299对其信号进行放大转换, 得到的数字量经过低功耗MCU压缩^[13], 通过低功耗蓝牙或USB传输至软件系统.MCU实时计算各通道阻抗, 超过软件阻抗阈值后, 在每个万向抱紧件上配置有警示灯实时提醒主试, 被试各通道电极的接触情况.

图 8 信号流程图

Fig. 8 Signal flow illustration

下载: 全尺寸图片幻灯片

SignBrain设备在开启蓝牙传输时, 最大采样率支持976 S/s, 采用USB传输最大采样率支持2 kS/s.

1.4 接口设计

SignBrain设备硬件上可通过Type-C连接外设控制器, 记录实验过程中不同trial的关键信息, 关键信息标签位与该时刻16通道脑电数据组合, 通过无线蓝牙传输至软件系统, 无线延时小于10 ms. 同样上位机软件系统也可通过USB或者UART等接口与外设控制器进行连接, 完成Marker记录. 对于软件系统中集成的实验范式, 可在软件中直接记录.

1.5 核心参数国内外对比

相比于32/64/128导联脑电设备, SignBrain爪形设备并不针对高通量脑电采集. 在设计之初, 设备主要考虑是能更加轻便、佩戴更加舒适, 使其适合长时间使用. SignBrain设备采用盐水作为导电介质, 实验前后无需洗头, 在临床门诊或高效采集中, 可穿戴、轻便、快速使用变得更为重要. 表1列举了美国Emotiv、美国CGX设备、奥地利g.tec g.Nautilus与SignBrain设备的对比. 其中美国CGX设备、美国Emotiv为类似可穿戴爪形结构设计的脑电设备, 奥地利g.tec g.Nautilus为脑电帽结构脑电设备.

表 1 SignBrain与其他厂商便携脑电设备的技术指标对比

Table 1 Comparison of technical specificities between SignBrain and the existing portable EEG devices

对比参数	SignBrain	Emotiv^[14]	美国 CGX^[15]	奥地利 g.tec^[16]
通道数量	16	14	20	16
采样频率 (Hz)	≥976	256	500	500
A/D位数	24	16	24	24
A/D分辨率 (μV)	0.53	0.51	—	0.53
阻抗	> 500 MΩ	—	—	> 100 MΩ
共模抑制比 (dB)	≥ 111	—	—	—
噪声 (uV)	≤ 1 μVpp	—	< 1 μVrms	< 0.6 μVrms
结构设计	爪形	爪形	爪形	脑电帽
重量	160 g	170 g	526 g	140 g

下载: 导出CSV

| 显示表格

可以发现, SignBrain的输入阻抗明显高于其他三个厂商, 对于微弱的脑电信号, 输入阻抗越高, 有助于更好地捕捉微弱信号, 数据采集更加准确, 并且可以减少外部噪声对信号的影响, 从而提高信号的信噪比. 同时SignBrain具有比其他厂商更高的采样率, 高采样率能够捕捉到快速变化的脑电信号, 如尖峰和快速振荡. 这对于分析脑电图的动态特征非常重要, 高采样率提供更好的时间分辨率, 在时间上相近的脑电活动可以被分开, 从而更好地分析不同事件相关的脑电活动. 佩戴方式上, 设备具有除脑电帽结构外, 比其他厂商更轻的重量, 对于快速应用的脑电场景中非常重要.

为进一步实际对比信号质量上差异, 在使用相同被试同一天进行静息态闭眼脑电信号采集, 采用Matlab对信号完成0.5 ~ 40 Hz带通滤波及FFT后, 得到频谱图发现SignBrain设备的底噪要好于Emotiv (图9). 采用相同处理方法, 将其与德国BP非穿戴设备的信号进行对比 (图10), 发现SignBrain设备的信号同样出色, 其中BP设备的具体型号为: BrainAmp DC^[17], 它是一款高性能的32导联脑电数据采集台式设备, 广泛应用于神经科学、心理学及临床研究. 表2为SignBrain设备与德国BP设备技术指标对比. 从指标上看, SignBrain设备的性能相比于德国BP高性能台式脑电设备并没有显著优势, 但通过频谱图上观察, SignBrain设备底噪的控制可以达到BP台式脑电设备的水平.

图 9 SignBrain与Emotiv的频谱图对比

Fig. 9 Spectrogram comparison between SignBrain and Emotiv

下载: 全尺寸图片幻灯片

图 10 SignBrain与BP的频谱图对比

Fig. 10 Spectrogram comparison between SignBrain and BP

下载: 全尺寸图片幻灯片

表 2 SignBrain设备与德国BP设备技术指标对比

Table 2 Comparison of technical specificities between SignBrain device and German BP device

对比参数	SignBrain	BrainAmp DC
通道数量	16	32
采样频率 (Hz)	976	5000
A/D分辨率 (μV)	0.53	0.1
输入动态范围μV(Vpp)	±17578	±16384
阻抗	实时阻抗	非实时阻抗
共模抑制比 (dB)	≥ 111	≥110
噪声 (uV)	≤ 1	< 1
电极材质	Ag/AgCL	Ag/AgCL

下载: 导出CSV

| 显示表格

2. SignBrain在临床医院验证

SignBrain设备已在各大医院及相关单位完成超过400例关于正常人、情绪检测、抑郁患者等数据的采集, 这些数据中包括静息态和任务态数据. 实际应用测试场景如图11所示.

图 11 实际应用测试图

Fig. 11 Practical application test

下载: 全尺寸图片幻灯片

前期主要包含两家医院, 北京大学第六医院和中国中医科学院广安门医院进行抑郁患者脑电数据的采集, 截至目前, 设备在两家医院采集的总人数为353例. 下面将简单介绍医院所采集脑电数据, 验证设备在实地不同人群的适用性及本设备所采集的脑电信号在抑郁症分类效果, 进而证明设备的易用性及应用价值.

图12为医院随机抽取5名男性被试实验照片截图, 图13为医院随机抽取5名女性被试实验照片截图. 通过视频记录发现10名被试在头发数量上也存在明显差异.

图 12 5名男性被试

Fig. 12 5 male participants

下载: 全尺寸图片幻灯片

图 13 5名女性被试

Fig. 13 5 female participants

下载: 全尺寸图片幻灯片

分别对男性女性被试相同通道的阻抗的均值进行对比, 以柱状图显示可以发现 (图14): (a) 10名被试中, 男性和女性佩戴设备后接触阻抗均小于18 kΩ; (b)从视频记录中看出女性的头发较长情况下, 接触阻抗仍小于18 kΩ. 同时我们对男性和女性被试所有通道阻抗值做双样本T检验, p值为0.93, 说明两个性别组无显著差异, 进一步证明设备接触阻抗不受性别影响.

图 14 不同性别各通道阻抗均值比较

Fig. 14 Comparison of the average impedance of each channel for different genders

下载: 全尺寸图片幻灯片

上述主要原因在于SignBrain设备采用爪形结构, 佩戴过程中, 每一个分支均可以很轻松穿过头发达到目标导联区域, 不会将整体头发压住, 这也是调节接触阻抗时间明显短于脑电帽的原因.

2.1 设备信号在抑郁分类上效果

通过抑郁自评量表SDS和具体得分人数统计, 发现353例医院的数据中有297人具有抑郁倾向, 无抑郁倾向56人.

同时, 对SignBrain设备采集的脑电数据进行整理、数据预处理, 采用深度学习DCAAN^[18][19]模型对处理后的数据集进行训练, 区分正常、轻度、中重度抑郁, 得到结果如表3所示, 发现其准确率达到70.61% (相关结果待发表), 在自有数据集实现了临床抑郁症的精准诊断.

表 3 不同模型在SignBrain数据集上进行抑郁分类效果

Table 3 The performance of different models in classifying depression on the SignBrain dataset

Model	Acc	Pre	Rec	F1-score
DAN (Long et al. , 2015)	74.36	58.92	60.19	58.91
DANN (Ganin et al. , 2016)	74.37	65.92	63.09	62.88
ADA (Haeusser et al. , 2017)	80.47	69.37	68.81	67.95
DCANN (Ours)	82.73	70.61	69.42	68.67

下载: 导出CSV

| 显示表格

上述数据集为SignBrain设备在临床医院两年内的数据积累, 其中大部分为门诊数据, 考虑到门诊问诊时间较短, 设备在研制设计初期, 初衷是可以快速完成可穿戴, 高质量完成脑电数据采集. 并且SignBrain设备结合上位机软件可以在数据采集后快速回看并上传至远端服务器, 这一特点大大减少脑电实验过程中需要科研人员协同医生完成脑电采集的时间, 有问题的实验数据也可快速提醒纠正. 实现了在临床医院, 患者无需过多等待, 医生无需过多操作, 科研人员远端回溯分析的特点.

3. 设备想象写字及高频视觉诱发验证

本节将简单介绍SignBrain设备在想象写字及高频视觉诱发实验范式上的完整测试和验证, 通过在线脑机交互实验, 验证设备在脑机交互领域上的可行性.

3.1 设备验证: 想象写字实验

3.1.1 想象写字方案设计

实验过程中, 参与者全程闭眼. 按照预设的26个字母的顺序, 在脑海中依次想象每个字母的书写轨迹. 为确保数据的精确性, 每个字母的想象被控制在约3秒的时间内. 在开始想象和结束想象这两个时间点, 参与者需要打Marker标记, 用于准确提取每个字母的脑电数据标签. 为了保持受试者的注意力和精力, 两次字母想象之间设置了1秒的休息时间. 这段时间允许大脑短暂的放松, 为下一次想象做好准备. 实验范式见图15.

图 15 想象写字范式图

Fig. 15 Experiment for imaginary writing recognition

下载: 全尺寸图片幻灯片

3.1.2 预处理及网络结构

对原始数据进行“滤波-降采样-重参考-标准化”等预处理流程. 首先提取0.5 ~ 40 Hz频段的信号, 去除50 Hz工频, 降采样到250 Hz. 其次对数据进行重参考, 消除电极位置的影响. 最终数据的维度是(16, 750, n). 维度16是信号通道的数量; 维度750 = 250*3是数据采样点的数量, 每一个字母是3 s的数据, 降采样到250 Hz; 维度n是预处理后提取了多少个频段的信号, 本次提取了n = 3个频段的信号, 分别为: Alpha波段、Beta波段以及包含多个波段的0.5 Hz ~ 40 Hz.

网络结构主要由时间维度卷积网络和域对抗全连接层构成. 首先将(16, 750, n = 3)维度的数据串联构造新维度(48, 750, 1)数据, 在此维度上完成不同视野卷积核在时间维度750上进行卷积, 其中特征提取器的损失函数见式 (1).

$$ {L}_{predict}=(1-\mathrm{e})H({\mathit{y}}_{\mathit{t}\mathit{u}\mathit{r}\mathit{e}}, \;{\mathit{y}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}})+eH ( {\mathit{y}}_{\mathit{s}\mathit{o}\mathit{f}\mathit{t}}, \;{\mathit{y}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}} )$$

(1)

式中$ {L}_{predict} $是最终损失, e为超参数 (强化模型对某些字母的关注), $ \left(1-\mathrm{e}\right)$为权重, $ H({\mathit{y}}_{\mathit{t}\mathit{u}\mathit{r}\mathit{e}}, \;{\mathit{y}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}})$是真实标签与预测标签之间的交叉熵, $ H $($ {\mathit{y}}_{\mathit{s}\mathit{o}\mathit{f}\mathit{t}}, \;{\mathit{y}}_{\mathit{p}\mathit{r}\mathit{e}\mathit{d}} $)是标签平滑的向量与预测标签之间的交叉熵, $ {\mathit{y}}_{\mathit{s}\mathit{o}\mathit{f}\mathit{t}} $是进行标签平滑的向量.

加入域对抗模块^[19]将梯度反转层置于特征提取器的末端, 通过计算源域特征和目标域特征的最大均值差异, 模型可以更加关注不同域之间的特征分布差异问题, 提升模型对跨时间脑电信号的泛化性. 最大均值差异是基于再生核希尔伯特空间的距离来度量两个分布的差异. 具体损失函数见式 (2). 特征提取器朝着域不变特征的方向更新其权重, 提取同一类别不同时间的相似特征, 提升脑电信号跨时间分类.

$$ \begin{split}{L}_{MMD}=\;&D\left({M}^{s}, \;{M}^{t}\right)=\|\frac{1}{a}\sum _{i=1}^{a}H\left({M}_{i}^{s}\right)\\&-\frac{1}{b}\sum _{i=1}^{b}H\left({M}_{i}^{t}\right)\| \end{split}$$

(2)

式中H是希尔伯特空间变换; $ {M}^{s} $是源域的特征; $ {M}^{t} $是目标域的特征.

3.1.3 模型训练及结果评估

使用SignBrain设备采集800组数据, 模型以80%作为训练集, 10%作为验证集, 10%作为测试集的配比进行训练. 训练好的模型经过优化, 将其封装成独立想象测试软件, SignBrain设备实时采集的脑电想象信号蓝牙上传至软件, 经预处理, 模型计算后, 可在0.5s内输出被试者想象的字母, 在对26个字母进行整体测试发现平均准确度在86.5%, 具体结果见图16. 被试使用SignBrain设备真实想象字母照片如图17所示.

图 16 26字母分类测试结果

Fig. 16 26 letter recognition results

下载: 全尺寸图片幻灯片

图 17 被试使用设备想象字母

Fig. 17 Subjects used the device to imagine letters writing

下载: 全尺寸图片幻灯片

3.1.4 BP设备想象写字信号对比

为增加SignBrain与商用BP设备在想象写字信号结果上的对比, 采用相同范式、相同被试、相同模型训练, 使用Brain Products采集的900组数据和使用SignBrain设备采集的800组数据, 分别进行模型训练、结果如图18所示, 实验1为BP分类准确率为81.1%, 实验2为SignBrain设备的分类准确率为86.5%, 由实验1和2可见脑电信号的实际运用场景中, SignBrain设备和Brain Products产品均能达到较好的实验效果.

图 18 采用BP设备与SignBrain设备想象写字实验模型训练结果

Fig. 18 Training results of the imaginary writing recognition using BP device and SignBrain device

下载: 全尺寸图片幻灯片

3.2 设备验证: 高频视觉诱发实验

3.2.1 高频视觉诱发实验设计

高频视觉诱发实验要求测试人员佩戴好SignBrain设备, 所有电极的接触阻抗在10 kΩ左右, 屏幕上依次按照A到Z的顺序闪烁26个字母和一个删除键和一个空格键, 频率从8 Hz到26.9 Hz, 以0.7 Hz频率间隔为步长, 1.5 s标记一次, 实验3 s, 休息0.5 s. 该实验模式下测试两组. 所有测试数据经过预处理、分段后、使用eCCA^[20] (expanded Canonical Correlation Analysis)从诱发的信号片段中提取出干净的SSVEP成分, 抑制噪声和共模干扰, 最终将整个实验步骤方法集成到在线视觉诱发软件中进行在线测试.

3.2.2 高频视觉诱发实验结果

为展示设备在视觉通道上采集诱发信号的准确度, 选用O1采集的视觉诱发信号, 从8 Hz到26.9 Hz中随机选择10.8、13.6、15、20.6、22.7、24.1这些频率点进行FFT, 可发现设备完整保留了诱发信号的频率 (图19). 同时为对比视觉诱发频率与刺激频率的误差, 将上述5个频率点挑选出计算误差 (表4), 发现设备在被试视觉皮层O1通道诱发频率与刺激频率误差小于10%, 诱发频率的误差完全满足步长0.7 Hz的设计. 视觉诱发展示链接: https://pan.cstcloud.cn/s/dKHCh5IoQLU.

图 19 SignBrain设备6个频率点视觉诱发O1电极记录脑电信号FFT图

Fig. 19 Demonstrations for the six frequency evoked signals by SSVEP using SignBrain Device

下载: 全尺寸图片幻灯片

表 4 视觉皮层O1通道诱发频率与刺激频率的误差

Table 4 The error between the induced frequency of the O1 channel and the stimulus frequency

刺激频率 (Hz)	10.8	13.6	15	20.6	22.7	24.1
实际诱发频率 (Hz)	10.75	13.58	15.03	20.65	22.64	24.07
误差 (%)	7.1	2.8	4.3	7.1	8.6	4.3

下载: 导出CSV

| 显示表格

经实验测试, 整体26个字母在该频率区间进行视觉诱发均可以超过95%的准确率. 该实验范式无需在特定环境下进行, 被试佩戴好SignBrain设备即可开始视觉诱发实验, 脑电数据通过低延时的蓝牙发送至上位机软件并在极短时间内完成信号的比对分析, 最终呈现在显示器上. 由此说明, SignBrain设备在脑电信号采集中表现了较高的信噪比, 通信传输延时率低, 可长时间稳定采集脑电信号.

4. 讨论

SignBrain设备结构上设计为爪形, 易于佩戴, 在很多场景及范式上可以高效的完成脑电采集. 数据通过无线蓝牙发送至配套软件系统, 不受具体场地约束.

设备已在多家医院及单位累计采集超过400例脑电数据, 并对相应任务的数据集分析, 均达到预期实验结果. 本文介绍的26个字母想象范式是受试者在家中佩戴设备进行测试. 字母想象范式实验在侵入式脑电设备上应用较多, 由于微型电极直接接触神经元放电, 信号质量非常高, 可以很轻松取得较高的准确率. 但在非侵入设备上, 外部噪声及头皮外获取信号的影响, 会大大影响信号质量, 非侵入设备难度极大, 需要设备具有更高的分辨率, 信噪比及共模抑制比, 实时分辨的效果还取决于信号传输及处理过程的时间. SignBrain设备采集想象字母的数据, 对其分类, 26个字母中有22个字母的准确率达到80%以上, 有12个字母的准确率达到100%. 可见, SignBrain设备利用16通道电极即可完成想象写字, 具有较高的信噪比.

5. 结论

本文介绍了团队自主研发的SignBrain可穿戴脑电设备的特点和优势. SignBrain设备轻巧便携, 爪形结构设计确保电极始终贴合头皮, 信号稳定, 使用盐水电极便于在各种场景快速佩戴. 设备具有高信噪比和低噪声, 并在想象字母范式和视觉诱发实验中验证了信号质量, 为后续研究脑电活动信号变化提供了可靠技术支持. 该设备针对快速可穿戴用途设计, 同时脑电性能和安全性均通过医疗级检验, 可在临床疾病诊断、脑机交互、特殊场景的脑认知检测, 以及特殊移动场景下实现快速应用. 关于设备的开发和应用的讨论网站: www. SignBrain.cn, 欢迎访问交流.

致谢

SignBrain项目研发及应用工作的快速推进特别感谢四川省科技计划项目2022YFSY0014 (2022-2024)、中国科学院先导专项(Grant No. QYZDJ-SSW-SMC019)、科技部科技创新2030 - 脑科学与类脑智能项目(Grant No. 2021ZD0200200)支持与资助, 同时也感谢SignBrain项目团队及相关实验医院的协助.

图 1 PFC启发Transformer结构

Fig. 1 PFC-inspired Transformer structure

下载: 全尺寸图片幻灯片

图 2 PFC生物功能启发生物功能模型与神经网络架构

Fig. 2 PFC biofunctional-inspired biofunctional model with neural network architecture

下载: 全尺寸图片幻灯片

图 3 PFC与类脑智能相互促进、共同进步

Fig. 3 PFC and brain-like intelligence promote each other and progress together

下载: 全尺寸图片幻灯片

图 4 PFC与Transformer注意力相关模型架构^{[14, 23]}

Fig. 4 PFC and Transformer attention-related model architecture^{[14, 23]}

下载: 全尺寸图片幻灯片

图 5 PFC与Transformer以注意力机制为媒介相互启发

Fig. 5 PFC and Transformer inspire each other through the medium of the attention mechanism

下载: 全尺寸图片幻灯片

图 6 标量相对位置编码 (SRPE) 原理

Fig. 6 Principle of Scalar Relative Position Encoding(SRPE)

下载: 全尺寸图片幻灯片

图 7 PFC生物编码过程启发Transformer位置编码

Fig. 7 PFC biological coding process inspired Transformer position coding

下载: 全尺寸图片幻灯片

图 8 不同模态下EEG分类的多尺度卷积Transformer模型^[63]

Fig. 8 Multi-scale convolutional Transformer model for EEG classification in different modalities^[63]

下载: 全尺寸图片幻灯片

图 9 PFC多感觉融合与多模态Transformer逻辑结构图

Fig. 9 PFC multisensory fusion with multimodal Transformer logic structure diagram

下载: 全尺寸图片幻灯片

参考文献(134)

[1]	CECCARELLI F, FERRUCCI L, LONDEI F, et al. Static and dynamic coding in distinct cell types during associative learning in the prefrontal cortex. Nature Communications, 2023, 14(1): 8325. doi: 10.1038/s41467-023-43712-2
[2]	WATAKABE A, SKIBBE H, NAKAE K, et al. Local and long-distance organization of prefrontal cortex circuits in the marmoset brain [J]. Neuron, 2023, 111 (14): 2258-73. e10.
[3]	PASSINGHAM R E, LAU H. Do we understand the prefrontal cortex?. Brain Structure and Function, 2023, 228(5): 1095−105.
[4]	TRAPP N T, BRUSS J E, MANZEL K, et al. Large-scale lesion symptom mapping of depression identifies brain regions for risk and resilience. Brain, 2023, 146(4): 1672−85. doi: 10.1093/brain/awac361
[5]	CHAFEE M V, HEILBRONNER S R. Prefrontal cortex. Current Biology, 2022, 32(8): R346−R51. doi: 10.1016/j.cub.2022.02.071
[6]	MILLER E K, COHEN J D. An integrative theory of prefrontal cortex function. Annual Review Of Neuroscience, 2001, 24: 167−202. doi: 10.1146/annurev.neuro.24.1.167
[7]	DIEHL G W, REDISH A D. Differential processing of decision information in subregions of rodent medial prefrontal cortex [J]. Elife, 2023, 12 .
[8]	WANG J X, KURTH-NELSON Z, KUMARAN D, et al. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 2018, 21(6): 860−8. doi: 10.1038/s41593-018-0147-8
[9]	ALEXANDER W H, BROWN J W. Medial prefrontal cortex as an action-outcome predictor. Nature Neuroscience, 2011, 14(10): 1338−44. doi: 10.1038/nn.2921
[10]	RIKHYE R V, GILRA A, HALASSA M M. Thalamic regulation of switching between cortical representations enables cognitive flexibility. Nature Neuroscience, 2018, 21(12): 1753−63. doi: 10.1038/s41593-018-0269-z
[11]	GISIGER T, BOUKADOUM M. Mechanisms Gating the Flow of Information in the Cortex: What They Might Look Like and What Their Uses may be. Frontiers In Computational Neuroscience, 2011, 5: 1.
[12]	JOHNSTON K, LEVIN H M, KOVAL M J, EVERLING S. Top-down control-signal dynamics in anterior cingulate and prefrontal cortex neurons following task switching. Neuron, 2007, 53(3): 453−62. doi: 10.1016/j.neuron.2006.12.023
[13]	TSUDA B, TYE K M, SIEGELMANN H T, SEJNOWSKI T J. A modeling framework for adaptive lifelong learning with transfer and savings through gating in the prefrontal cortex. Proceedings Of The National Academy Of Sciences, 2020, 117(47): 29872−82. doi: 10.1073/pnas.2009591117
[14]	WANG Z, ZHANG J, ZHANG X, et al. Transformer model for functional near-infrared spectroscopy classification. IEEE Journal Of Biomedical And Health Informatics, 2022, 26(6): 2559−69. doi: 10.1109/JBHI.2022.3140531
[15]	CHOI S R, LEE M. Transformer architecture and attention mechanisms in genome data analysis: a comprehensive review. Biology, 2023, 12(7): 1033. doi: 10.3390/biology12071033
[16]	LI Q, ZHUANG Y. An efficient image-guided-based 3D point cloud moving object segmentation with transformer-attention in autonomous driving. International Journal Of Applied Earth Observation And Geoinformation, 2023, 123: 103488. doi: 10.1016/j.jag.2023.103488
[17]	BRUS J, HENG J A, BELIAEVA V, et al. Causal phase-dependent control of non-spatial attention in human prefrontal cortex. Nature Human Behaviour, 2024, 8(4): 743−57. doi: 10.1038/s41562-024-01820-z
[18]	BICHOT N P, HEARD M T, DEGENNARO E M, DESIMONE R. A Source for Feature-Based Attention in the Prefrontal Cortex. Neuron, 2015, 88(4): 832−44. doi: 10.1016/j.neuron.2015.10.001
[19]	HUANG L, WANG J, HE Q, et al. A source for category-induced global effects of feature-based attention in human prefrontal cortex. Cell Reports, 2023, 42(9): 113080. doi: 10.1016/j.celrep.2023.113080
[20]	ZHAO M, XU D, GAO T. From Cognition to Computation: A Comparative Review of Human Attention and Transformer Architectures [J]. arXiv preprint arXiv: 240701548, 2024.
[21]	KUMAR S, SUMERS T R, YAMAKOSHI T, et al. Shared functional specialization in transformer-based language models and the human brain. Nature Communications, 2024, 15(1): 5523. doi: 10.1038/s41467-024-49173-5
[22]	MULLER L, CHURCHLAND P S, SEJNOWSKI T J. Transformers and Cortical Waves: Encoders for Pulling In Context Across Time [J]. arXiv preprint arXiv: 240114267, 2024.
[23]	HUANG H, LI R, QIAO X, et al. Attentional control influence habituation through modulation of connectivity patterns within the prefrontal cortex: Insights from stereo-EEG. Neuroimage, 2024, 294: 120640. doi: 10.1016/j.neuroimage.2024.120640
[24]	LI N, CHEN Y, LI W, et al. BViT: Broad attention-based vision transformer [J]. IEEE Transactions On Neural Networks And Learning Systems, 2023.
[25]	SHI Q, FAN J, WANG Z, ZHANG Z. Multimodal channel-wise attention transformer inspired by multisensory integration mechanisms of the brain. Pattern Recognition, 2022, 130: 108837. doi: 10.1016/j.patcog.2022.108837
[26]	GONG D, ZHANG H. Self-Attention Limits Working Memory Capacity of Transformer-Based Models [J]. arXiv preprint arXiv: 240910715, 2024.
[27]	MAITH O, SCHWARZ A, HAMKER F H. Optimal attention tuning in a neuro-computational model of the visual cortex-basal ganglia-prefrontal cortex loop. Neural Networks, 2021, 142: 534−47. doi: 10.1016/j.neunet.2021.07.008
[28]	SPITALE G, BILLER-ANDORNO N, GERMANI F. AI model GPT-3 (dis) informs us better than humans. Science Advances, 2023, 9(26): eadh1850. doi: 10.1126/sciadv.adh1850
[29]	BROWN T, MANN B, RYDER N, et al. Language models are few-shot learners. Advances In Neural Information Processing Systems, 2020, 33: 1877−901.
[30]	ZHANG S, ROLLER S, GOYAL N, et al. Opt: Open pre-trained transformer language models [J]. arXiv preprint arXiv: 220501068, 2022.
[31]	YUE F, KO T. An Investigation of Positional Encoding in Transformer-based End-to-end Speech Recognition [Z]. In: Proceedings of the 12th International Symposium on Chinese Spoken Language Processing (ISCSLP). 2021: 1-5.10. 1109/iscslp49672.2021. 9362093
[32]	KAZEMNEJAD A, PADHI I, NATESAN RAMAMURTHY K, et al. The impact of positional encoding on length generalization in transformers [J]. Advances In Neural Information Processing Systems, 2024, 36 .
[33]	CHOWDHERY A, NARANG S, DEVLIN J, et al. Palm: Scaling language modeling with pathways. Journal Of Machine Learning Research, 2023, 24(240): 1−113.
[34]	ZHANG R, HAN J, LIU C, et al. Llama-adapter: Efficient fine-tuning of language models with zero-init attention [J]. arXiv preprint arXiv: 230316199, 2023.
[35]	TOUVRON H, LAVRIL T, IZACARD G, et al. Llama: Open and efficient foundation language models [J]. arXiv preprint arXiv: 230213971, 2023.
[36]	WU J, ZHANG R, MAO Y, CHEN J. On scalar embedding of relative positions in attention models; In: Proceedings of the AAAI Conference on Artificial Intelligence, F, 2021, 35(16): 14050-14057 [C].
[37]	WALLIS J D, ANDERSON K C, MILLER E K. Single neurons in prefrontal cortex encode abstract rules. Nature, 2001, 411(6840): 953−6. doi: 10.1038/35082081
[38]	OOTA S R, ARORA J, ROWTULA V, et al. Visio-linguistic brain encoding [J]. arXiv preprint arXiv: 220408261, 2022.
[39]	BOCINCOVA A, BUSCHMAN T J, STOKES M G, MANOHAR S G. Neural signature of flexible coding in prefrontal cortex. Proceedings Of The National Academy Of Science USA, 2022, 119(40): e2200400119. doi: 10.1073/pnas.2200400119
[40]	ZHANG K, HAO W, YU X, SHAO T. An interpretable image classification model Combining a fuzzy neural network with a variational autoencoder inspired by the human brain. Information Sciences, 2024, 661: 119885. doi: 10.1016/j.ins.2023.119885
[41]	AOI M C, MANTE V, PILLOW J W. Prefrontal cortex exhibits multidimensional dynamic encoding during decision-making. Nature Neuroscience, 2020, 23(11): 1410−20. doi: 10.1038/s41593-020-0696-5
[42]	ZHANG Z, GONG X. Positional label for self-supervised vision transformer; In: Proceedings of the AAAI Conference on Artificial Intelligence, F, 2023, 37(3): 3516-3524 [C].
[43]	CAUCHETEUX C, GRAMFORT A, KING J R. Evidence of a predictive coding hierarchy in the human brain listening to speech. Nature Human Behavaviour, 2023, 7(3): 430−41. doi: 10.1038/s41562-022-01516-2
[44]	BUSCH A, ROUSSY M, LUNA R, et al. Neuronal activation sequences in lateral prefrontal cortex encode visuospatial working memory during virtual navigation. Nature Communications, 2024, 15(1): 4471. doi: 10.1038/s41467-024-48664-9
[45]	LABAIEN J, IDé T, CHEN P-Y, et al. Diagnostic spatio-temporal transformer with faithful encoding [J]. Knowledge-Based Systems, 2023, 274 .
[46]	DEIHIM A, ALONSO E, APOSTOLOPOULOU D. STTRE: A Spatio-Temporal Transformer with Relative Embeddings for multivariate time series forecasting. Neural Networks, 2023, 168: 549−59. doi: 10.1016/j.neunet.2023.09.039
[47]	MA Y, WANG R. Relative-position embedding based spatially and temporally decoupled Transformer for action recognition [J]. Pattern Recognition, 2024, 145 .
[48]	COEN P, SIT T P H, WELLS M J, et al. Mouse frontal cortex mediates additive multisensory decisions [J]. Neuron, 2023, 111 (15): 2432-47 e13.
[49]	FERRARI A, NOPPENEY U. Attention controls multisensory perception via two distinct mechanisms at different levels of the cortical hierarchy. PLoS Biology, 2021, 19(11): e3001465. doi: 10.1371/journal.pbio.3001465
[50]	MIHALIK A, NOPPENEY U. Causal inference in audiovisual perception. Journal Of Neuroscience, 2020, 40(34): 6600−12. doi: 10.1523/JNEUROSCI.0051-20.2020
[51]	KANG K, ROSENKRANZ R, KARAN K, et al. Congruence-based contextual plausibility modulates cortical activity during vibrotactile perception in virtual multisensory environments. Communications Biology, 2022, 5(1): 1360. doi: 10.1038/s42003-022-04318-4
[52]	CAO Y, SUMMERFIELD C, PARK H, et al. Causal inference in the multisensory brain [J]. Neuron, 2019, 102 (5): 1076-87. e8.
[53]	GIESSING C, THIEL C M, STEPHAN K E, et al. Visuospatial attention: how to measure effects of infrequent, unattended events in a blocked stimulus design. Neuroimage, 2004, 23(4): 1370−81. doi: 10.1016/j.neuroimage.2004.08.008
[54]	ZHENG Q, ZHOU L, GU Y. Temporal synchrony effects of optic flow and vestibular inputs on multisensory heading perception [J]. Cell Reports, 2021, 37 (7).
[55]	LIANG P P, ZADEH A, MORENCY L-P. Foundations & Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions. ACM Computing Surveys, 2024, 56(10): 1−42.
[56]	KLEMEN J, CHAMBERS C D. Current perspectives and methods in studying neural mechanisms of multisensory interactions. Neuroscience & Biobehavioral Reviews, 2012, 36(1): 111−33.
[57]	PARASKEVOPOULOS G, GEORGIOU E, POTAMIANOS A. Mmlatch: Bottom-Up Top-Down Fusion For Multimodal Sentiment Analysis [Z]. In: Proceedings of the 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2022: 4573-7.10. 1109/icassp43922.2022. 9746418
[58]	SUN H, LIU J, CHEN Y-W, LIN L. Modality-invariant temporal representation learning for multimodal sentiment classification. Information Fusion, 2023, 91: 504−14. doi: 10.1016/j.inffus.2022.10.031
[59]	WANG Z, WAN Z, WAN X. Transmodality: An end2end fusion method with transformer for multimodal sentiment analysis; In: Proceedings of the web conference 2020, F, 2020: 2514-2520 [C].
[60]	YU J, CHEN K, XIA R. Hierarchical interactive multimodal transformer for aspect-based multimodal sentiment analysis. IEEE Transactions On Affective Computing, 2022, 14(3): 1966−78.
[61]	HUANG J, ZHOU J, TANG Z, et al. TMBL: Transformer-based multimodal binding learning model for multimodal sentiment analysis [J]. Knowledge-Based Systems, 2024, 285 .
[62]	YANG D, LIU Y, HUANG C, et al. Target and source modality co-reinforcement for emotion understanding from asynchronous multimodal sequences [J]. Knowledge-Based Systems, 2023, 265 .
[63]	AHN H J, LEE D H, JEONG J H, LEE S W. Multiscale Convolutional Transformer for EEG Classification of Mental Imagery in Different Modalities [J]. IEEE Transactions On Neural Systems And Rehabilitation Engineering, 2022, 31 : 646-656. PP.
[64]	LI J, CHEN N, ZHU H, et al. Incongruity-aware multimodal physiology signals fusion for emotion recognition [J]. Information Fusion, 2024, 105 .
[65]	ASIF M, GUPTA A, ADITYA A, et al. Brain Multi-Region Information Fusion using Attentional Transformer for EEG Based Affective Computing [Z]. In: Proceedings of the 20th India Council International Conference (INDICON). 2023: 771-5.10. 1109/indicon59947.2023. 10440791
[66]	CHEN Z, HAN Y, MA Z, et al. A prefrontal-thalamic circuit encodes social information for social recognition. Nature Communications, 2024, 15(1): 1036. doi: 10.1038/s41467-024-45376-y
[67]	YU J, LI J, YU Z, HUANG Q. Multimodal Transformer With Multi-View Visual Representation for Image Captioning. IEEE Transactions On Circuits And Systems For Video Technology, 2020, 30(12): 4467−80. doi: 10.1109/TCSVT.2019.2947482
[68]	HU B, GUAN Z-H, CHEN G, CHEN C P. Neuroscience and network dynamics toward brain-inspired intelligence. IEEE Transactions on Cybernetics, 2021, 52(10): 10214−27.
[69]	SUCHOLUTSKY I, MUTTENTHALER L, WELLER A, et al. Getting aligned on representational alignment [J]. arXiv preprint arXiv: 231013018, 2023.
[70]	CHERSONI E, SANTUS E, HUANG C-R, LENCI A. Decoding word embeddings with brain-based semantic features. Computational Linguistics, 2021, 47(3): 663−98. doi: 10.1162/coli_a_00412
[71]	TONEVA M, WEHBE L. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) [J]. Advances in Neural Information Processing Systems, 2019, 32 .
[72]	YU S, GU C, HUANG K, LI P. Predicting the next sentence (not word) in large language models: What model-brain alignment tells us about discourse comprehension. Science Advances, 2024, 10(21): eadn7744. doi: 10.1126/sciadv.adn7744
[73]	CAMBRIA E, DAS D, BANDYOPADHYAY S, FERACO A. Affective computing and sentiment analysis [J]. A Practical Guide To Sentiment Analysis, 2017: 1-10 .
[74]	MISHRA A, DEY K, BHATTACHARYYA P. Learning cognitive features from gaze data for sentiment and sarcasm classification using convolutional neural network; In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), F, 2017 [C].
[75]	ZHANG Z, WU C, CHEN H, CHEN H. CogAware: Cognition-Aware framework for sentiment analysis with textual representations [J]. Knowledge-Based Systems, 2024, 299 .
[76]	MONTEJO-RáEZ A, MOLINA-GONZáLEZ M D, JIMéNEZ-ZAFRA S M, et al. A survey on detecting mental disorders with natural language processing: Literature review, trends and challenges. Computer Science Review, 2024, 53: 100654. doi: 10.1016/j.cosrev.2024.100654
[77]	RAMACHANDRAN G, YANG R. CortexCompile: Harnessing Cortical-Inspired Architectures for Enhanced Multi-Agent NLP Code Synthesis [J]. arXiv preprint arXiv: 240902938, 2024.
[78]	LI Z, ZHAO B, ZHANG G, DANG J. Brain network features differentiate intentions from different emotional expressions of the same text; In: Proceedings of the 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), F, 2023 [C]. IEEE.
[79]	SQUIRES M, TAO X, ELANGOVAN S, et al. Deep learning and machine learning in psychiatry: a survey of current progress in depression detection, diagnosis and treatment. Brain Informatics, 2023, 10(1): 10. doi: 10.1186/s40708-023-00188-6
[80]	SONG G, HUANG D, XIAO Z. A study of multilingual toxic text detection approaches under imbalanced sample distribution. Information, 2021, 12(5): 205. doi: 10.3390/info12050205
[81]	DOSOVITSKIY A, BEYER L, KOLESNIKOV A, et al. An image is worth 16x16 words: Transformers for image recognition at scale [J]. arXiv preprint arXiv: 201011929, 2020.
[82]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need [J]. Advances In Neural Information Processing Systems, 2017, 30 .
[83]	BI Y, ABROL A, JIA S, et al. Gray Matters: An Efficient Vision Transformer GAN Framework for Predicting Functional Network Connectivity Biomarkers from Brain Structure [J]. BioRxiv, 2024: 2024.01. 11.575307.
[84]	DONG S, GONG Y, SHI J, et al. Brain Cognition-Inspired Dual-Pathway CNN Architecture for Image Classification. IEEE Transactions On Neural Networks And Learning Systems, 2024, 35(7): 9900−14. doi: 10.1109/TNNLS.2023.3237962
[85]	LIU L, WANG F, ZHOU K, et al. Perceptual integration rapidly activates dorsal visual pathway to guide local processing in early visual areas. PLoS Biology, 2017, 15(11): e2003646. doi: 10.1371/journal.pbio.2003646
[86]	BAR M. The proactive brain: using analogies and associations to generate predictions. Trends In Cognitive Sciences, 2007, 11(7): 280−9. doi: 10.1016/j.tics.2007.05.005
[87]	BARAM A B, MULLER T H, NILI H, et al. Entorhinal and ventromedial prefrontal cortices abstract and generalize the structure of reinforcement learning problems [J]. Neuron, 2021, 109 (4): 713-23 e7.
[88]	VAN HOLSTEIN M, FLORESCO S B. Dissociable roles for the ventral and dorsal medial prefrontal cortex in cue-guided risk/reward decision making. Neuropsychopharmacology, 2020, 45(4): 683−93. doi: 10.1038/s41386-019-0557-7
[89]	AVERBECK B, O'DOHERTY J P. Reinforcement-learning in fronto-striatal circuits. Neuropsychopharmacology, 2022, 47(1): 147−62. doi: 10.1038/s41386-021-01108-0
[90]	HU S, SHEN L, ZHANG Y, et al. On Transforming Reinforcement Learning With Transformers: The Development Trajectory [J]. IEEE Transactions On Pattern Analysis And Machine Intelligence, 2024, PP.
[91]	ZHANG Y, JIA M, CHEN T, et al. A neuroergonomics model for evaluating nuclear power plants operators’ performance under heat stress driven by ECG time-frequency spectrums and fNIRS prefrontal cortex network: A CNN-GAT fusion model. Advanced Engineering Informatics, 2024, 62: 102563. doi: 10.1016/j.aei.2024.102563
[92]	LAW C-K, KOLLING N, CHAN C C, CHAU B K. Frontopolar cortex represents complex features and decision value during choice between environments [J]. Cell Reports, 2023, 42 (6).
[93]	LEE J, JUNG M, LUSTIG N, LEE J H. Neural representations of the perception of handwritten digits and visual objects from a convolutional neural network compared to humans. Human Brain Mapping, 2023, 44(5): 2018−38. doi: 10.1002/hbm.26189
[94]	VISWANATHAN K A, MYLAVARAPU G, CHEN K, THOMAS J P. A Study of Prefrontal Cortex Task Switching Using Spiking Neural Networks; In: Proceedings of the 12th International Conference on Advanced Computational Intelligence (ICACI), F, 2020 [C]. IEEE.
[95]	LI B-Z, PUN S H, FENG W, et al. A spiking neural network model mimicking the olfactory cortex for handwritten digit recognition; In: Proceedings of the 9th International IEEE/EMBS Conference on Neural Engineering (NER), F, 2019 [C]. IEEE.
[96]	HYAFIL A, SUMMERFIELD C, KOECHLIN E. Two mechanisms for task switching in the prefrontal cortex. Journal Of Neuroscience, 2009, 29(16): 5135−42. doi: 10.1523/JNEUROSCI.2828-08.2009
[97]	KUSHLEYEVA Y, SALVUCCI D D, LEE F J. Deciding when to switch tasks in time-critical multitasking. Cognitive Systems Research, 2005, 6(1): 41−9. doi: 10.1016/j.cogsys.2004.09.005
[98]	BRASS M, VON CRAMON D Y. The role of the frontal cortex in task preparation. Cerebral Cortex, 2002, 12(9): 908−14. doi: 10.1093/cercor/12.9.908
[99]	WEI Q, HAN L, ZHANG T. Learning and Controlling Multiscale Dynamics in Spiking Neural Networks Using Recursive Least Square Modifications [J]. IEEE Transactions On Cybernetics, 2024, PP.
[100]	DEMIR A, KOIKE-AKINO T, WANG Y, et al. EEG-GNN: Graph neural networks for classification of electroencephalogram (EEG) signals; proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), F, 2021 [C]. IEEE.
[101]	BALAJI S S, PARHI K K. Classifying Subjects with PFC Lesions from Healthy Controls during Working Memory Encoding via Graph Convolutional Networks [Z]. In: Proceedings of the 11th International IEEE/EMBS Conference on Neural Engineering (NER). 2023: 1-4.10. 1109/ner52421.2023. 10123793
[102]	YANG Y, YE C, MA T. A deep connectome learning network using graph convolution for connectome-disease association study. Neural Networks, 2023, 164: 91−104. doi: 10.1016/j.neunet.2023.04.025
[103]	ACHTERBERG J, AKARCA D, STROUSE D J, et al. Spatially embedded recurrent neural networks reveal widespread links between structural and functional neuroscience findings. Nature Machine Intelligence, 2023, 5(12): 1369−81. doi: 10.1038/s42256-023-00748-9
[104]	JENSEN K T, HENNEQUIN G, MATTAR M G. A recurrent network model of planning explains hippocampal replay and human behavior [J]. Nature Neuroscience, 2024: 1-9 .
[105]	PRATIWI M. Comparative Analysis of Brain Waves for EEG-Based Depression Detection in the Prefrontal Cortex Lobe using LSTM; In: Proceedings of the 7th International Conference on New Media Studies (CONMEDIA), F, 2023 [C]. IEEE.
[106]	PRATIWI M. EEG-Based Depression Detection in the Prefrontal Cortex Lobe using mRMR Feature Selection and Bidirectional LSTM. Ultima Computing: Jurnal Sistem Komputer, 2023, 15(2): 71−8. doi: 10.31937/sk.v15i2.3426
[107]	SHARMA S, SHARMA S, ATHAIYA A. Activation functions in neural networks. Towards Data Sci, 2017, 6(12): 310−6.
[108]	JAGTAP A D, KAWAGUCHI K, KARNIADAKIS G E. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. Journal Of Computational Physics, 2020, 404: 109136. doi: 10.1016/j.jcp.2019.109136
[109]	ABBASI J, ANDERSEN P Ø. Physical activation functions (PAFs): an approach for more efficient induction of physics into physics-informed neural networks (PINNs). Neurocomputing, 2024, 608: 128352. doi: 10.1016/j.neucom.2024.128352
[110]	JAGTAP A D, KARNIADAKIS G E. How important are activation functions in regression and classification? A survey, performance comparison, and future directions [J]. Journal Of Machine Learning For Modeling And Computing, 2023, 4 (1).
[111]	MANOLA L, ROELOFSEN B, HOLSHEIMER J, et al. Modelling motor cortex stimulation for chronic pain control: electrical potential field, activating functions and responses of simple nerve fibre models. Medical And Biological Engineering And Computing, 2005, 43: 335−43. doi: 10.1007/BF02345810
[112]	STEINERBERGER S, WU H-T. Fundamental component enhancement via adaptive nonlinear activation functions. Applied And Computational Harmonic Analysis, 2023, 63: 135−43. doi: 10.1016/j.acha.2022.11.007
[113]	PAPPAS C, KOVAIOS S, MORALIS-PEGIOS M, et al. Programmable tanh-, elu-, sigmoid-, and sin-based nonlinear activation functions for neuromorphic photonics [J]. IEEE Journal Of Selected Topics In Quantum Electronics, 2023, 29 (6: Photonic Signal Processing): 1-10.
[114]	HA D, SCHMIDHUBER J. World models [J]. arXiv preprint arXiv: 180310122, 2018.
[115]	ESLAMI S A, JIMENEZ REZENDE D, BESSE F, et al. Neural scene representation and rendering. Science, 2018, 360(6394): 1204−10. doi: 10.1126/science.aar6170
[116]	YAMINS D L, HONG H, CADIEU C F, et al. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings Of The National Academy Of Sciences, 2014, 111(23): 8619−24. doi: 10.1073/pnas.1403112111
[117]	FRISTON K, MORAN R J, NAGAI Y, et al. World model learning and inference. Neural Networks, 2021, 144: 573−90. doi: 10.1016/j.neunet.2021.09.011
[118]	ROBINE J, HöFTMANN M, UELWER T, HARMELING S. Transformer-based world models are happy with 100k interactions [J]. arXiv preprint arXiv: 230307109, 2023.
[119]	MICHELI V, ALONSO E, FLEURET F. Transformers are sample-efficient world models [J]. arXiv preprint arXiv: 220900588, 2022.
[120]	CHEN C, WU Y-F, YOON J, AHN S. Transdreamer: Reinforcement learning with transformer world models [J]. arXiv preprint arXiv: 220209481, 2022.
[121]	ZHANG W, WANG G, SUN J, et al. STORM: Efficient stochastic transformer based world models for reinforcement learning [J]. Advances In Neural Information Processing Systems, 2024, 36 .
[122]	HAFNER D, PASUKONIS J, BA J, LILLICRAP T. Mastering diverse domains through world models [J]. arXiv preprint arXiv: 230104104, 2023.
[123]	HAFNER D, LILLICRAP T, NOROUZI M, BA J. Mastering atari with discrete world models [J]. arXiv preprint arXiv: 201002193, 2020.
[124]	BARTO A G, SUTTON R S, ANDERSON C W. Looking back on the actor–critic architecture. IEEE Transactions On Systems, Man, And Cybernetics: Systems, 2020, 51(1): 40−50.
[125]	KAISER L, BABAEIZADEH M, MILOS P, et al. Model-based reinforcement learning for atari [J]. arXiv preprint arXiv: 190300374, 2019.
[126]	MOERLAND T M, BROEKENS J, PLAAT A, JONKER C M. Model-based reinforcement learning: A survey. Foundations And Trends® In Machine Learning, 2023, 16(1): 1−118.
[127]	GU A, GOEL K, Ré C. Efficiently modeling long sequences with structured state spaces [J]. arXiv preprint arXiv: 211100396, 2021.
[128]	SMITH J T, WARRINGTON A, LINDERMAN S W. Simplified state space layers for sequence modeling [J]. arXiv preprint arXiv: 220804933, 2022.
[129]	DENG F, PARK J, AHN S. Facing off world model backbones: Rnns, transformers, and s4 [J]. Advances In Neural Information Processing Systems, 2024, 36 .
[130]	CHEN J, LI S E, TOMIZUKA M. Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning. IEEE Transactions On Intelligent Transportation Systems, 2021, 23(6): 5068−78.
[131]	HAFNER D, LILLICRAP T, BA J, NOROUZI M. Dream to control: Learning behaviors by latent imagination [J]. arXiv preprint arXiv: 191201603, 2019.
[132]	ZHANG Y, MU Y, YANG Y, et al. Steadily learn to drive with virtual memory [J]. arXiv preprint arXiv: 210208072, 2021.
[133]	GAO Z, MU Y, CHEN C, et al. Enhance sample efficiency and robustness of end-to-end urban autonomous driving via semantic masked world model [J]. IEEE Transactions On Intelligent Transportation Systems, 2024.
[134]	YU N, LV Z, YAN J, WANG Z. Spatial Cognition and Decision Model Based on Hippocampus-Prefrontal Cortex Interaction [Z]. 2023 China Automation Congress (CAC). 2023: 3754-9.10. 1109/cac59555.2023. 10450650.