Machine Translation Hallucination and Omission Detection via the Integration of Large Language Models and Dual-Stream Feature Networks
-
摘要: 机器翻译幻觉是指译文引入与源文无关的信息, 漏译是指译文遗漏源文关键信息. 针对现有基于模型内部状态的方法依赖额外内部信息、基于外部工具的方法与任务目标不完全一致的问题, 提出结合大语言模型与双流特征网络的Hybrid-HOD. 该方法构建由深度语义特征流和多视角语义对齐特征流组成的异构双流特征, 前者建模源文、机器译文、伪参考译文与回译之间的全局语义一致性, 后者刻画逻辑蕴含、语义相似度和字符级一致性等细粒度对齐信息. 模型通过注意力门控机制融合异构双流特征, 并结合不确定性多任务学习策略优化参数, 在推理阶段与大语言模型直接评分进行线性加权融合, 实现对幻觉与漏译的有效检测. 公开数据集HalOmi上的实验表明, 该方法在句子级严重度排序与二分类任务中均优于基线.Abstract: Machine translation hallucination refers to the generation of information unrelated to the source text, whereas omission refers to the failure to translate key information from the source text. Existing methods based on internal model states rely on additional information generated during translation inference, while external-tool-based methods suffer from a mismatch between their training objectives and the target task. To overcome these limitations, Hybrid-HOD is proposed as a machine translation hallucination and omission detection framework that integrates large language models with a dual-stream feature network. The framework constructs heterogeneous dual-stream representations consisting of a deep semantic feature stream and a multi-view semantic alignment feature stream. The former models the global semantic consistency among the source text, machine-translated text, pseudo-reference translation, and back-translation, while the latter captures fine-grained alignment information, including logical entailment, semantic similarity, and character-level consistency. An attention-gated fusion mechanism is employed to integrate the heterogeneous dual-stream representations, and an uncertainty-based multi-task learning strategy is adopted for parameter optimization. During inference, the direct scores generated by the large language model are linearly fused with the predictions of the dual-stream feature network to improve hallucination and omission detection. Experimental results on the public HalOmi dataset demonstrate that Hybrid-HOD consistently outperforms baseline methods on both sentence-level severity ranking and binary classification tasks.1)
1 https://www.deepseek.com/2)2 https://chat.openai.com/ -
图 1 结合大语言模型与双流特征网络的机器翻译幻觉与漏译检测方法架构图
Fig. 1 Architecture of machine translation hallucination and omission detection method combining large language models with dual-stream feature networks
表 1 不同方法在HalOmi数据集上的PCP指标
Table 1 PCP results of different methods on the HalOmi dataset
方法 幻觉PCP 漏译PCP 高资源 低资源 高资源 低资源 Seq-Logprob 0.835 0.784 0.770 0.768 COMET-Kiwi 0.867 0.654 0.849 0.667 BLASER2-QE 0.884 0.829 0.870 0.844 OTTAWA 0.842 0.741 0.755 0.775 GPT-4o-mini 0.843 0.666 0.487 0.664 DeepSeek-V3.1 0.873 0.784 0.762 0.779 GPT-5.1 0.866 0.764 0.805 0.760 Hybrid-HOD 0.919 0.812 0.848 0.820 Hybrid-HOD
(DeepSeek-V3.1)0.937 0.834 0.882 0.848 注: PCP为无量纲比例, 取值范围为$ [0,\;1] $; 粗体表示最优结果, 下划线表示次优结果; Hybrid-HOD(DeepSeek-V3.1)表示在推理阶段结合DeepSeek-V3.1模型. 
下载: 导出CSV
表 2 不同方法在HalOmi数据集上的幻觉二分类结果
Table 2 Binary classification results for hallucination detection of different methods on the HalOmi dataset
方法 高资源 低资源 MCC F1 AUC MCC F1 AUC Seq-Logprob 0.583 0.719 0.842 0.413 0.717 0.780 COMET-Kiwi 0.612 0.741 0.875 0.216 0.729 0.670 STARE 0.655 0.718 0.894 0.472 0.696 0.807 BLASER2-QE 0.615 0.733 0.887 0.461 0.746 0.819 GPT-4o-mini 0.644 0.765 0.887 0.352 0.763 0.771 DeepSeek-V3.1 0.661 0.780 0.886 0.449 0.758 0.800 GPT-5.1 0.634 0.732 0.887 0.420 0.617 0.780 Hybrid-HOD (DeepSeek-V3.1) 0.734 0.812 0.937 0.477 0.786 0.834 注: 粗体表示最优结果, 下划线表示次优结果.
下载: 导出CSV
表 3 不同方法在HalOmi数据集上的漏译二分类结果
Table 3 Binary classification results for omission detection of different methods on the HalOmi dataset
方法 高资源 低资源 MCC F1 AUC MCC F1 AUC Seq-Logprob 0.400 0.610 0.756 0.375 0.710 0.757 COMET-Kiwi 0.574 0.745 0.844 0.129 0.779 0.687 STARE 0.512 0.733 0.864 0.455 0.730 0.830 BLASER2-QE 0.552 0.751 0.860 0.472 0.761 0.843 GPT-4o-mini 0.148 0.540 0.633 0.308 0.460 0.638 DeepSeek-V3.1 0.485 0.717 0.828 0.516 0.812 0.828 GPT-5.1 0.592 0.738 0.856 0.470 0.683 0.813 Hybrid-HOD (DeepSeek-V3.1) 0.624 0.755 0.887 0.549 0.817 0.853 注: 表1—3均按照高资源语言和低资源语言分别统计; Hybrid-HOD表示仅使用双流特征网络, 不融合LLMs的直接评分; Hybrid-HOD(DeepSeek-V3.1)表示在推理阶段进一步融合DeepSeek-V3.1的直接评分. 粗体表示最优结果, 下划线表示次优结果.
下载: 导出CSV
表 4 不同组件与策略对模型性能的影响
Table 4 Impact of different components and strategies on model performance
配置 幻觉 漏译 MCC PCP MCC PCP 移除幻觉辅助头 N/A 0.878 0.554 0.860 移除多维投影 0.588 0.878 0.569 0.860 移除注意力门控模块 0.550 0.870 0.565 0.846 移除交互张量 0.603 0.862 0.587 0.855 移除深度语义特征流 0.567 0.873 0.509 0.854 移除多视角对齐特征流 0.591 0.856 0.545 0.842 Hybrid-HOD 0.621 0.881 0.574 0.864 注: 表中各项指标均在完整测试集上计算, 未区分高资源语言和低资源语言; N/A表示该配置下不产生相应的分类结果.
下载: 导出CSV
表 5 Hybrid-HOD结合不同LLMs直接评分的性能
Table 5 Performance of Hybrid-HOD combined with direct scoring from different LLMs
推理配置 幻觉PCP 漏译PCP 高资源 低资源 高资源 低资源 DeepSeek-V3.1 0.873 0.784 0.762 0.779 GPT-4o-mini 0.843 0.666 0.487 0.664 GPT-5.1 0.866 0.764 0.805 0.760 Hybrid-HOD 0.919 0.812 0.848 0.820 + DeepSeek-V3.1 0.937 0.834 0.882 0.848 + GPT-4o-mini 0.924 0.823 0.845 0.820 + GPT-5.1 0.927 0.829 0.884 0.844 注: “+模型”表示将Hybrid-HOD的预测结果与相应大语言模型的直接评分结果进行融合; 粗体表示最优结果, 下划线表示次优结果.
下载: 导出CSV
表 6 机器翻译错误案例分析
Table 6 Analysis of machine translation error cases
样例1 样例2 src Привет, Дим!!! Por ello, los permisos de travesía para toda fecha de inicio dentro del mes de mayo estarán disponibles a partir del 1 de enero. mt What＇s going on? For this reason, travel authorisations for any start date within May will be available from 1 January. hall. s/p 0.75 / 0.93 −0.01 / 0.19 omit. s/p 0.75 / 0.86 0.01 / 0.25 注: src表示源文本, mt表示机器译文; hall. s/p表示幻觉严重度评分与幻觉概率; omit. s/p表示漏译严重度评分与漏译概率.
下载: 导出CSV
-
[1] Pang J, Ye F, Wong D F, Yu D, Shi S, Tu Z, et al. Salute the classic: revisiting challenges of machine translation in the age of large language models. Transactions of the Association for Computational Linguistics, 2025, 13: 73−95 doi: 10.1162/tacl_a_00730 [2] Ataman D, Birch A, Habash N, Federico M, Koehn P, Cho K. Machine translation in the era of large language models: a survey of historical and emerging problems. Information, 2025, 16(9): 723 doi: 10.3390/info16090723 [3] 李洪政, 冯冲, 黄河燕. 稀缺资源语言神经网络机器翻译研究综述. 自动化学报, 2021, 47(6): 1217−1231 doi: 10.16383/j.aas.c200103Li Hong-Zheng, Feng Chong, Huang He-Yan. A survey on low-resource neural machine translation. Acta Automatica Sinica, 2021, 47(6): 1217−1231 doi: 10.16383/j.aas.c200103 [4] Dale D, Voita E, Lam J, Hansanti P, Ropers C, Kalbassi E, et al. HalOmi: a manually annotated benchmark for multilingual hallucination and omission detection in machine translation. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023. 638-653 [5] Zhang Y, Li Y, Cui L, Cai D, Liu L, Fu T, et al. Siren's song in the AI ocean: a survey on hallucination in large language models. Computational Linguistics, 2025, 51(4): 1373−1418 doi: 10.1162/COLI.a.16 [6] Kadavath S, Conerly T, Askell A, Henighan T, Drain D, Perez E, et al. Language models (mostly) know what they know. arXiv preprint arXiv: 2207.05221, 2022 [7] Meng K, Bau D, Andonian A, Belinkov Y. Locating and editing factual associations in GPT. In: Proceedings of Advances in Neural Information Processing Systems. New Orleans, LA, USA: Curran Associates, 2022. 17359-17372 [8] Guerreiro N M, Voita E, Martins A. Looking for a needle in a haystack: a comprehensive study of hallucinations in neural machine translation. In: Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics. Dubrovnik, Croatia: Association for Computational Linguistics, 2023. 1059-1075 [9] Wang X, Wei J, Schuurmans D, Le Q, Chi E, Narang S, et al. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv: 2203.11171, 2022 [10] Ji Z, Lee N, Frieske R, Yu T, Su D, Xu Y, et al. Survey of hallucination in natural language generation. ACM Computing Surveys, 2023, 55(12): Article No. 248, 1−38 [11] Conneau A, Rinott R, Lample G, Williams A, Bowman S R, Schwenk H, et al. XNLI: evaluating cross-lingual sentence representations. In: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, 2018. 2475-2485 [12] Guerreiro N M, Colombo P, Piantanida P, Martins A. Optimal transport for unsupervised hallucination detection in neural machine translation. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics, 2023. 13766-13784 [13] 黄鑫, 张家俊, 宗成庆. 基于跨模态实体信息融合的神经机器翻译方法. 自动化学报, 2023, 49(6): 1170−1180 doi: 10.16383/j.aas.c220230Huang Xin, Zhang Jia-Jun, Zong Cheng-Qing. Neural machine translation method based on cross-modal entity information fusion. Acta Automatica Sinica, 2023, 49(6): 1170−1180 doi: 10.16383/j.aas.c220230 [14] 钟茂生, 刘会珠, 匡江玲, 严婷. DeepCom-GCN: 融入控制流结构信息的代码注释生成模型. 江西师范大学学报(自然科学版), 2025, 49(1): 27−36 doi: 10.16357/j.cnki.issn1000-5862.2025.01.05Zhong Mao-Sheng, Liu Hui-Zhu, Kuang Jiang-Ling, Yan Ting. DeepCom-GCN: integrated control-flow structural information code comment generation model. Jiangxi Normal University Journal (Natural Science Edition), 2025, 49(1): 27−36 doi: 10.16357/j.cnki.issn1000-5862.2025.01.05 [15] 于志强, 余正涛, 黄于欣, 郭军军, 高盛祥. 基于变分信息瓶颈的半监督神经机器翻译. 自动化学报, 2022, 48(7): 1678−1689 doi: 10.16383/j.aas.c190477Yu Zhi-Qiang, Yu Zheng-Tao, Huang Yu-Xin, Guo Jun-Jun, Gao Sheng-Xiang. Improving semi-supervised neural machine translation with variational information bottleneck. Acta Automatica Sinica, 2022, 48(7): 1678−1689 doi: 10.16383/j.aas.c190477 [16] Ferrando J, Gállego G I, Alastruey B, Escolano C, Costa-jussà M R. Towards opening the black box of neural machine translation: source and target interpretations of the transformer. In: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022. 8756-8769 [17] Dale D, Voita E, Barrault L, Costa-jussà M R. Detecting and mitigating hallucinations in machine translation: model internal workings alone do well, sentence similarity even better. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics, 2023. 36-50 [18] Farquhar S, Kossen J, Kuhn L, Gal Y. Detecting hallucinations in large language models using semantic entropy. Nature, 2024, 630(8017): 625−630 doi: 10.1038/s41586-024-07421-0 [19] Brown P F, Della Pietra S A, Della Pietra V J, Mercer R L. The mathematics of statistical machine translation: parameter estimation. Computational Linguistics, 1993, 19(2): 263−311 [20] Zenkel T, Wuebker J, DeNero J. End-to-end neural word alignment outperforms GIZA++. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020. 1605-1617 [21] Huang C, Ghaddar A, Kobyzev I, Rezagholizadeh M, Zaiane O, Chen B. OTTAWA: optimal TransporT adaptive word aligner for hallucination and omission translation errors detection. In: Findings of the Association for Computational Linguistics: ACL 2024. Bangkok, Thailand: Association for Computational Linguistics, 2024. 6322-6334 [22] 翟煜锦, 李培芸, 项青宇, 李茂西, 裘白莲, 钟茂生, 等. 基于QE的机器翻译重排序方法研究. 江西师范大学学报(自然科学版), 2020, 44(1): 46−50 doi: 10.16357/j.cnki.issn1000-5862.2020.01.08Zhai Yu-Jin, Li Pei-Yun, Xiang Qing-Yu, Li Mao-Xi, Qiu Bai-Lian, Zhong Mao-Sheng, et al. Study on machine translation reranking method based on QE. Jiangxi Normal University Journal (Natural Science Edition), 2020, 44(1): 46−50 doi: 10.16357/j.cnki.issn1000-5862.2020.01.08 [23] Rei R, Treviso M, Guerreiro N M, et al. CometKiwi: IST-Unbabel 2022 submission for the quality estimation shared task. In: Proceedings of the Seventh Conference on Machine Translation (WMT). Abu Dhabi, United Arab Emirates (Hybrid): Association for Computational Linguistics, 2022. 634-645 [24] Dale D, Costa-jussà M R. BLASER 2.0: a metric for evaluation and quality estimation of massively multilingual speech and text translation. In: Findings of the Association for Computational Linguistics: EMNLP 2024. Miami, Florida, USA: Association for Computational Linguistics, 2024. 16075-16085 [25] Kocmi T, Federmann C. GEMBA-MQM: detecting translation quality error spans with GPT-4. In: Proceedings of the Eighth Conference on Machine Translation. Singapore: Association for Computational Linguistics, 2023. 768-775 [26] Conneau A, Khandelwal K, Goyal N, Chaudhary V, Wenzek G, Guzmán F, et al. Unsupervised cross-lingual representation learning at scale. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020. 8440-8451 [27] Reimers N, Gurevych I. Sentence-BERT: sentence embeddings using Siamese BERT-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics, 2019. 3982-3992 [28] Sanh V, Debut L, Chaumond J, Wolf T. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 1910.01108, 2019 [29] Liu Y, Ott M, Goyal N, Du J, Joshi M, Chen D, et al. RoBERTa: a robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1907.11692, 2019 [30] Reimers N, Gurevych I. Making monolingual sentence embeddings multilingual using knowledge distillation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Online: Association for Computational Linguistics, 2020. 4512-4525 [31] Kendall A, Gal Y, Cipolla R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Salt Lake City, USA: IEEE Computer Society, 2018. 7482-7491 [32] Himmi A, Staerman G, Picot M, Colombo P, Guerreiro N M. Enhanced hallucination detection in neural machine translation through simple detector aggregation. In: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Miami, Florida, USA: Association for Computational Linguistics, 2024. 18573-18583 -
下载:
计量
- 文章访问数: 6
- HTML全文浏览量: 7
- 被引次数: 0
