多尺度视觉语义增强的多模态命名实体识别方法

王海荣; 徐玺; 王彤; 陈芳萍

doi:10.16383/j.aas.c230573

多尺度视觉语义增强的多模态命名实体识别方法

doi: 10.16383/j.aas.c230573

王海荣^{1, 2,},
徐玺^1,,
王彤^1,,
陈芳萍^1,

1.
北方民族大学计算机科学与工程学院银川 750021
2.
北方民族大学图像图形智能处理国家民委重点实验室银川 750021

基金项目: 宁夏自然科学基金项目(2023AAC03316), 宁夏回族自治区教育厅高等学校科学研究重点项目 (NYG2022051)资助

详细信息

作者简介:
王海荣：北方民族大学教授, 2015年获得东北大学博士学位. 主要研究方向为大数据知识工程与智能信息处理. 本文通信作者. E-mail: wanghr@nun.edu.cn

徐玺：北方民族大学计算机科学与工程学院硕士研究生. 主要研究方向为多模态信息抽取. E-mail: 20217403@stu.nmu.edu.cn

王彤：北方民族大学计算机科学与工程学院硕士研究生. 主要研究方向为多模态信息抽取. E-mail: is_wangtong@163.com

陈芳萍：北方民族大学计算机科学与工程学院硕士研究生. 主要研究方向为多模态信息抽取. E-mail: 17393213357@163.com

计量
- 文章访问数: 75
- HTML全文浏览量: 12
- 被引次数: 0
出版历程
- 收稿日期: 2023-09-13
- 录用日期: 2024-02-22
- 网络出版日期: 2024-04-01

Multi-Scale Visual Semantic Enhancement for Multi-Modal Named Entity Recognition Method

WANG Hai-Rong^{1, 2
,},
XU Xi^1
,,
WANG Tong^1
,,
CHEN Fang-Ping^1
,

1.
School of Computer Science and Engineering, North Minzu University, Yinchuan 750021
2.
The Key Laboratory of Images & Graphics Intelligent Processing of State Ethnic Affairs Commission, North Minzu University, Yinchuan 750021

Funds: Supported by Natural Science Foundation of Ningxia Province(2023AAC03316), and Key research project of Ningxia Education Department (NYG2022051)

More Information

Author Bio:
WANG Hai-Rong　Professor at North Minzu University. She received her Ph.D. degrees from Northeastern University in 2015. Her research interest covers Big Data Knowledge Engineering and Intelligent Information Processing. Corresponding author of this paper

XU Xi　Master student at the School of Computer Science and Engineering, North Minzu University. His research interest covers Multimodal information extraction

WANG Tong　Master student at the School of Computer Science and Engineering, North Minzu University. Her research interest covers Multimodal information extraction

CHEN Fang-Ping　Master student at the School of Computer Science and Engineering, North Minzu University. Her research interest covers Multimodal information extraction

摘要

摘要: 为了解决多模态命名实体识别(Multi-modal named entity recognition, MNER) 方法研究中存在的图像特征语义缺失以及多模态表示语义约束较弱等问题, 提出了多尺度视觉语义增强的多模态命名实体识别方法. 该方法提取多种视觉特征用于补全图像语义; 挖掘文本特征与多种视觉特征间的语义交互关系, 生成多尺度视觉语义特征并进行融合, 得到多尺度视觉语义增强的多模态文本表示; 使用视觉实体分类器对多尺度视觉语义特征解码, 实现视觉特征的语义一致性约束; 调用多任务标签解码器挖掘多模态文本表示和文本特征的细粒度语义, 通过联合解码来解决语义偏差问题, 从而进一步提高命名实体识别准确度. 为验证该方法的有效性, 在Twitter-2015和Twitter-2017数据集上进行实验, 并与HvpNet、MNER-QG和RGCN等10种方法进行对比, 该方法的平均F1值分别提升了0.85%和1.45%.
- 多模态命名实体识别 /
- 多任务学习 /
- 多模态融合 /
- Transformer
Abstract: To address the issues of semantic loss in image features and weak semantic constraints in multi-modal representations encountered in the research of multi-modal named entity recognition (MNER) methods, multi-scale visual semantic enhancement for multi-modal named entity recognition method is proposed. After supplementing image semantics by extracting multiple visual features, the semantic interaction and feature fusion between text features and various visual features are explored through a multi-modal feature fusion module. This process outputs multi-scale visual semantic features, text features, and multi-scale visual semantic-enhanced multi-modal text representations. The visual entity classifier is used to decode multi-scale visual semantic features to learn the semantic consistency between various visual features. The multi-task decoder is invoked to mine the fine-grained semantic representation in multi-modal text repre-sentation and text features, and carry out joint decoding to solve the semantic bias problem, thereby further improving the accuracy of named entity recognition. To verify the effectiveness of the method, experiments were carried out on Twitter-2015 and Twitter-2017 respectively, and compared with 10 methods such as HvpNet, MNER-QG, RGCN, etc. The average F1 values of the MSVSE on the two datasets have increased by 0.85% and 1.45%, respectively.
- Multi-modal named entity recognition(MNER) /
- multitasking learning /
- multi-modal fusion /
- transformer

HTML全文

图 1 MSVSE模型框架

Fig. 1 The framework of MSVSE model

下载: 全尺寸图片幻灯片

图 2 多模态特征融合

Fig. 2 The multi-modal feature fusion module

下载: 全尺寸图片幻灯片

图 3 多任务标签解码器

Fig. 3 The multi-tasking label decoder

下载: 全尺寸图片幻灯片

图 4 在Twitter-2015的视觉实体分类性能比较

Fig. 4 Performance comparison of visual entity classification on Twitter-2015

下载: 全尺寸图片幻灯片

图 5 在Twitter-2017的视觉实体分类性能比较

Fig. 5 Performance comparison of visual entity classification on Twitter-2017

下载: 全尺寸图片幻灯片

表 1 数据集上方法性能比较(%)

Table 1 Performance comparison of method on Twitter dataset (%)

方法	Twitter-2015					Twitter-2017
方法	PER	LOC	ORG	MISC	F1	PER	LOC	ORG	MISC	F1
MSB	86.44	77.16	52.91	36.05	73.47	—	—	—	—	84.32
MAF	84.67	81.18	63.35	41.82	73.42	91.51	85.80	85.10	68.79	86.25
UMGF	84.26	83.17	62.45	42.42	74.85	91.92	85.22	83.13	69.83	85.51
M3S	86.05	81.32	62.97	41.36	75.03	92.73	84.81	82.49	69.53	86.06
UMT	85.24	81.58	63.03	39.45	73.41	91.56	84.73	82.24	70.10	85.31
UAMer	84.95	81.28	61.41	38.34	73.10	90.49	81.52	82.09	64.32	84.90
VAE	85.82	81.56	63.20	43.67	75.07	91.96	81.89	84.13	74.07	86.37
MNER-QG	85.68	81.42	63.62	41.53	74.94	93.17	86.02	84.64	71.83	87.25
RGCN	86.36	82.08	60.78	41.56	75.00	92.86	86.10	84.05	72.38	87.11
HvpNet	85.74	81.78	61.92	40.81	74.33	92.28	84.81	84.37	65.20	85.80
MSVSE	86.72	81.63	64.08	38.91	75.11	93.24	85.96	85.22	70.00	87.34
–HvpNet	0.98	–0.15	2.16	–1.90	0.78	0.96	1.15	0.85	4.80	1.54

下载: 导出CSV

表 2 模型结构消融实验(%)

Table 2 Structural ablation experiments for the model(%)

方法	Twitter-2015					Twitter-2017
方法	PER	LOC	ORG	MISC	F1	PER	LOC	ORG	MISC	F1
MSVSE(本文)	86.72	81.63	64.08	38.91	75.11	93.24	85.96	85.22	70.00	87.34
w/o自注意力机制	86.49	81.20	63.21	41.56	74.83	93.05	86.52	84.37	67.34	86.79
w/o相似度	86.33	81.59	63.15	40.84	74.91	92.94	86.59	84.07	68.24	86.75
w/o自注意力机制+相似度	86.80	81.38	63.32	39.62	74.67	92.97	85.87	84.41	67.96	86.67
w/o多任务标签解码器	86.49	81.78	62.68	37.60	74.69	92.98	84.83	85.02	71.66	87.14
w/o视觉实体分类器	86.52	81.64	63.06	39.89	74.79	93.37	84.83	85.82	66.24	86.92

下载: 导出CSV

表 3 联合编码器中视觉特征消融实验(%)

Table 3 Visual feature ablation experiments in the joint encoder(%)

文本	视觉标签	图像描述	Twitter-2015					Twitter-2017
文本	视觉标签	图像描述	PER	LOC	ORG	MISC	F1	PER	LOC	ORG	MISC	F1
$ \checkmark$	—	$ \checkmark$	86.72	81.63	64.08	38.91	75.11	93.24	85.96	85.22	70.00	87.34
$ \checkmark$	—	—	86.76	81.68	61.21	39.46	74.73	92.95	86.20	84.60	70.82	87.11
$ \checkmark$	$ \checkmark$	—	86.87	81.74	63.72	37.80	74.87	93.03	85.71	84.43	71.71	87.16
$ \checkmark$	$ \checkmark$	$ \checkmark$	86.51	81.85	62.20	38.36	74.72	93.73	85.96	84.62	70.97	87.38

下载: 导出CSV

表 4 多尺度视觉语义前缀中视觉特征消融实验(%)

Table 4 Visual feature ablation experiments in multi-scale visual semantic prefixes (%)

区域视觉特征	视觉标签	图像描述	Twitter-2015					Twitter-2017
区域视觉特征	视觉标签	图像描述	PER	LOC	ORG	MISC	F1	PER	LOC	ORG	MISC	F1
$ \checkmark$	$ \checkmark$	$ \checkmark$	86.72	81.63	64.08	38.91	75.11	93.24	85.96	85.22	70.00	87.34
$ \checkmark$	—	—	86.25	81.93	63.99	38.23	74.76	93.16	84.83	85.47	69.10	87.13
$ \checkmark$	$ \checkmark$	—	86.56	81.60	64.01	38.59	74.93	93.02	85.79	85.97	68.67	87.28
$ \checkmark$	—	$ \checkmark$	86.87	81.79	63.36	38.68	74.98	92.94	86.52	85.14	68.94	87.14

下载: 导出CSV

表 5 单尺度视觉特征下方法性能对比(%)

Table 5 Performance comparison of methods under single scale visual feature (%)

方法	单尺度视觉特征	Twitter-2015(F1)	Twitter-2017(F1)
MAF	区域视觉特征	73.42	86.25
MSB	图像标签	73.47	84.32
ITA	视觉标签	75.18	85.67
ITA	5个视觉描述	75.17	85.75
ITA	光学字符识别	75.01	85.64
MSVSE	only区域视觉特征	74.84	86.75
MSVSE	only视觉标签	74.66	87.17
MSVSE	only视觉描述	74.56	87.23
MSVSE	w/o视觉前缀	74.89	87.08
MSVSE(本文)		75.11	87.34

下载: 导出CSV

表 6 不同学习率下方法性能对比(%)

Table 6 Performance comparison of methods under different learning rates (%)

学习率($ \times {{10}^{-5}}$)	1	2	3	4	5	6
Twitter-2015	73.4	75	75.1	74.8	74.6	74.5
Twitter-2017	87.1	86.8	87.3	87.5	87.2	87.3

下载: 导出CSV

表 7 参数量及时间效率对比

Table 7 Comparison of parameter number and time efficiency

方法	参数量(MB)	训练时间(s)	验证时间(s)
MSB	122.97	45.80	3.31
UMGF	191.32	314.42	18.73
MAF	136.09	103.39	6.37
ITA	122.97	65.40	4.69
UMT	148.10	156.73	8.59
HvpNet	143.34	70.36	9.34
MSVSE(本文)	119.27	75.81	7.03

下载: 导出CSV

表 8 基于预训练语言模型的MNER方法性能对比(%)

Table 8 Performance comparison of MNER method based on pre-trained language model (%)

方法	Twitter-2015	Twitter-2017
Glove-BiLSTM-CRF	69.15	79.37
BERT-CRF	71.81	83.44
BERT-large-CRF	73.53	86.81
XLMR-CRF	77.37	89.39
Prompting ChatGPT	79.33	91.43
MSVSE(本文)	75.11	87.34

下载: 导出CSV

参考文献(28)

[1]	Moon S, Neves L, Carvalho V. Multimodal named entity recognition for short social media posts. In: Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. New Orleans, USA: NAACL Press, 2018. 852-860
[2]	Lu D, Neves L, Carvalho V, Zhang N, Ji H. Visual attention model for name tagging in multimodal social media. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Melbourne, Australia: 2018, (1). 1990-1999
[3]	Asgari-Chenaghlu M, Farzinvash M R, FARZINVASH L, Balafar M A, Motamed C. CWI: A multimodal deep learning approach for named entity recognition from social media using character, word and image features. Neural Computing and Applications, 2022, 34(3): 1905−1922 doi: 10.1007/s00521-021-06488-4
[4]	Zhang Q, Fu J L, Liu X Y, Huang X J. Adaptive co-attention network for named entity recognition in tweets. In: Proceedings of the 32th AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. New Orleans, USA: AAAI Press, 2018. 5674−5681
[5]	Zheng C M, Wu Z W, Wang T, Cai Y, Li Q. Object-aware multimodal named entity recognition in social media Posts with adversarial learning. IEEE Transactions on Multimedia, 2020, 23: 2520−2532
[6]	Wu Z W, Zheng C M, Cai Y, Chen J Y, Leung H F, Li Q. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In: Proceedings of the 28th ACM International Conference on Multimedia. Seattle, USA: ACM, 2020. 1038−1046
[7]	Yu J F, Jiang J, Yang L, Xia R. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Virtual Event: 2020. 3342−3352
[8]	Xu B, Huang S Z, Sha C F, Wang H Y. MAF: A general matching and alignment framework for multimodal named entity recognition. In: Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining. Association for Computing Machinery, New York, USA: 2022. 1215−1223
[9]	Wang X W, Ye J B, Li Z X, Tian J F, Jiang Y, Yan M, et al. CAT-MNER: Multimodal named entity recognition with knowledge refined cross-modal attention. In: Proceedings of the IEEE International Conference on Multimedia and Expo. Taipei, China: 2022. 1−6
[10]	Zhang D, Wei S Z, Li S S, Wu H Q, Zhu Q M, Zhou G D. Multi-modal graph fusion for named entity recognition with targeted visual guidance. In: Proceedings of the AAAI Conference on Artificial Intelligence. Palo Alto, USA: AAAI Press, 2021. 35(16). 14347−14355
[11]	钟维幸, 王海荣, 王栋, 车淼. 多模态语义协同交互的图文联合命名实体识别方法. 广西科学, 2022, 29(04): 681−690 Zhong Wei-Xing, Wang Hai-Rong, Wang Dong, Che Miao. lmage-text joint named entity recognition method based on multi-modal semantic interaction. Guangxi Sciences, 2022, 29(04): 681−690
[12]	Yu T, Sun X, Yu H F, Li Y, Fu K. Hierarchical self-adaptation network for multimodal named entity recognition in social media. Neurocomputing, 2021(439): 12−21
[13]	Wang X Y, Gui M, Jiang Y, jia Z X, Bach N, Wang T, et al. ITA: Image-text alignments for multi-modal named entity recognition. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, USA: Association for Computational Linguistics, 2022. 3176−3189
[14]	Liu L P, Wang M L, Zhang M Z, Qing L B, He X H. UAMNer: Uncertainty aware multi-modal named entity recognition in social media posts. Appl Intell, 2022, 52(4): 4109−4125 doi: 10.1007/s10489-021-02546-5
[15]	李晓腾, 张盼盼, 勾智楠, 高凯. 基于多任务学习的多模态命名实体识别方法. 计算机工程, 2023, 49(04): 114−119 Li Xiao-Teng, Zhang Pan-Pan, Gou Zhi-Nan, Gao Kai. Multi-modal named entity recognition method based on multi-task learning. Computer Engineering, 2023, 49(04): 114−119
[16]	Wang J, Yang Y, Liu K Y, Zhu Z P, Liu X R. M3S: Scene graph driven multi-granularity multi-task learning for multi-modal NER. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2023(31): 111−120
[17]	Chen X, Zhang N Y, Li L, Yao Y Z, Deng S M, Tan C Q, et al. Good visual guidance make a better extractor: hierarchical visual prefix for multimodal entity and relation extraction. In: Proceedings of the Association for Computational Linguistics. Seattle, USA: Association for Computational Linguistics, 2022. 1607−1618
[18]	Jia M H Z, Shen L, Shen X, Liao L J, Chen M, He X D, et al. MNER-QG: An end-to-end mrc framework for multimodal named entity recognition with query grounding. AAAI, 2022, 37(7): 8032−8040
[19]	Sun L, Wang J Q, Su Y D, Weng F S, Sun Y X, Zheng Z W, et al. RIVA: A pre-trained tweet multimodal model based on text-image relation for multimodal ner. In: Proceedings of the 28th International Conference on Computational Linguistics. Virtual Event: 2022. 1852−1862
[20]	Sun L, Wang J Q, Zhang K, Su Y D, Weng F S. RpBERT: A Text-image relation propagation-based BERT model for multimodal ner. In: Proceedings of the AAAI Conference on Artificial Intelligence, Virtual Event: 2021. 35(15): 13860−13868
[21]	Xu B, Huang S, Du M, Wang H Y, Song H, Sha C F, et al. Different data, different modalities! reinforced data splitting for effective multimodal information extraction from social media posts. In: Proceedings of the 29th International Conference on Computational Linguistics. Virtual Event: 2022. 1855−1864
[22]	Zhao F, Li C H, Wu Z, Xing S Y, Dai X Y. Learning from different text-image pairs: a relation-enhanced graph convolutional network for multimodal ner. In: Proceedings of the 30th ACM International Conference on Multimedia, Association for Computing Machinery. New York, USA: 2022. 3983−3992
[23]	Zhou B H, Zhang Y, Song K H, Guo W Y, Zhao G Q, Wang W B, et al. A span-based multimodal variational autoencoder for semi-supervised multimodal named entity recognition. In: Proceedings of the Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022. 6293−6302
[24]	He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. In: Proceedings of the IEEE International Conference on Computer Vision. Venice, Italy: 2017. 2980−2988
[25]	Vinyals O, Toshev A, Bengio S, Erhan D. Show and tell: a neural image caption generator. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: 2015. 3156−3164
[26]	He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: 2016. 770−778
[27]	王海荣, 徐玺, 王彤, 荆博祥. 多模态命名实体识别方法研究进展. 多模态命名实体识别方法研究进展. 郑州大学学报(工学版), doi: 10.13705/j.issn.1671-6833 Wang Hai-Rong, Xu Xi, Wang Tong, Jing Bo-Xiang. Research progress of multimodal named entity recognition. Journal of Zhengzhou University (Engineering Science), doi: 10.13705/j.issn.1671-6833
[28]	Li J Y, Li H, Pan Z, Sun D, Wang J H, Zhang W K, et al. Prompting ChatGPT in MNER: Enhanced multimodal named entity recognition with auxiliary refined knowledge. In: Proceedings of the Association for Computational Linguistics: EMNLP 2023. Singapore: 2023. 2787−2802