Review on Tongue Image Segmentation Technologies for Traditional Chinese Medicine: Methodologies, Performances and Prospects
-
摘要: 中医舌诊的客观化、定量化研究是中医现代化发展中的重要课题. 数字化采集到的舌图像包括舌体及部分面部区域, 为了便于后续舌象自动分析, 需要首先将舌体部分从图像中分割出来, 分割效果将直接影响后续舌象特征分析的准确性. 基于传统方法的舌象分割技术虽然取得了很大进展, 但其性能仅能达到半自动分割, 对较难分割的图像往往需要借助人机交互来完成. 近年来, 深度学习技术在图像处理及计算机视觉等多个领域取得了突破, 其在图像语义分割任务中也取得了远超传统方法的进展. 基于深度学习的舌象分割技术已经基本实现了全自动的鲁棒分割. 本文首先从传统分割方法和基于深度学习的分割方法两方面对中医舌象分割技术发展中的主要方法进行综述; 其次, 采用我们收集的舌象数据库对典型的方法进行性能评估, 并对不同舌象分割方法的特点进行分析与讨论. 最后, 对中医舌图像分割方法潜在的发展方向进行了展望.Abstract: The objectification and quantitative analysis of tongue diagnosis is an important topic in the development of traditional Chinese medicine (TCM) modernization. The digitally acquired tongue images include the tongue and part of the face region. In order to facilitate the automatic analysis of the tongue image, the tongue needs to be segmented from the whole image, and the segmentation results will directly affect the accuracy of the tongue image feature analysis. Although traditional methods of tongue-image segmentation has made great progress, their performance can only achieve semi-automatic segmentation. There are images that difficult to segment perfectly without human-computer interaction. In recent years, with the breakthrough of deep learning technology in the field of image processing and computer vision, it has achieved far more performance than traditional methods in the semantic segmentation tasks. The deep-learning based tongue-image segmentation technologies have achieved fully automatic robust segmentation. This survey gives a detailed overview of the history, state of the art, and typical methods in this domain. Firstly, the typical segmentation methods are presented. Then, they are used for migration learning and network testing based on our self-built tongue image database. In addition, this paper analyzes the characteristics of these segmentation methods and obtains the advantages and disadvantages of them. Finally, this paper summarizes the methods of Chinese medicine tongue image segmentation, and discussed to the development direction.
-
Key words:
- Chinese medicine tongue image /
- semantic segmentation /
- transfer learning /
- performance evaluation
1) 本文责任编委 刘成林 -
表 1 不同舌象分割方法性能比较
Table 1 Comparison on performances of diffierent algorithms
网络名称 mIoU 预测时间(s) FCN8S 0.8322 7.4227 FCN16S 0.8718 7.4403 FCN32S 0.9272 7.5273 SegNet 0.9277 0.0014 Mask R-CNN 0.9361 0.1412 Deeplab V2 0.9678 0.0021 DeeplabV3 0.9810 4.9999×10-6 DeeplabV3+ 0.9818 2.9999×10-6 -
[1] 沈兰荪, 蔡轶珩, 张新峰. 中医舌象的采集与分析. 北京: 北京工业大学出版社, 2007.Shen Lan-Sun, Cai Yi-Heng, Zhang Xin-Feng. Capturation and Analysis of Chinese Medicine Tongue Images. Beijing: Beijing University of Technology Press, 2007. [2] 李国正, 史淼晶, 李福凤, 王忆勤. 舌体图像分割技术的实验分析与改进. 山东大学学报(工学版), 2010, 40(5): 87-95 https://www.cnki.com.cn/Article/CJFDTOTAL-SDGY201005017.htmLi Guo-Zheng, Shi Miao-Jing, Li Fu-Feng, Wang Yi-Qin. An empirical study on tongue image detection. Journal of Shandong University (Engineering Science), 2010, 40(5): 87-95 https://www.cnki.com.cn/Article/CJFDTOTAL-SDGY201005017.htm [3] Kim K H, Do J H, Ryu H, Kim J Y. Tongue diagnosis method for extraction of effective region and classification of tongue coating. In: Proceedings of the 1st Workshops on Image Processing Theory, Tools and Applications. Sousse, Tunisia: IEEE, 2008. 1-7 [4] 邬文隽, 马利庄, 肖学中. 一种结合亮度和粗糙度信息的舌像分割方法. 系统仿真学报, 2006, 18(S1): 374-376, 379 https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ2006S1115.htmWu Wen-Jun, Ma Li-Zhuang, Xiao Xue-Zhong. Method of tongue image segmentation based on luminance and roughness information. Journal of System Simulation, 2006, 18(S1): 374-376, 379 https://www.cnki.com.cn/Article/CJFDTOTAL-XTFZ2006S1115.htm [5] 张志顺, 刘勇. 基于动态阈值和修正模型的舌体提取算法. 计算机与现代化, 2014, (11): 49-52 doi: 10.3969/j.issn.1006-2475.2014.11.011Zhang Zhi-Shun, Liu Yong. Tongue extraction algorithm based on dynamic thresholds and correction model. Computer and Modernization, 2014, (11): 49-52 doi: 10.3969/j.issn.1006-2475.2014.11.011 [6] 赵忠旭, 王爱民, 沈兰荪. 基于数学形态学和HIS模型的彩色舌图像分割. 北京工业大学学报, 1999, 25(2): 67-71 https://www.cnki.com.cn/Article/CJFDTOTAL-BJGD199902011.htmZhao Zhong-Xu, Wang Ai-Min, Shen Lan-Sun. The color tongue image segmentation based on mathematical morphology and HIS model. Journal of Beijing Polytechnic University, 1999, 25(2): 67-71 https://www.cnki.com.cn/Article/CJFDTOTAL-BJGD199902011.htm [7] Chen L, Wang D Y, Liu Y Q, Gao X H, Shang H L. A novel automatic tongue image segmentation algorithm: Color enhancement method based on L*a*b* color space. In: Proceedings of the 2015 IEEE International Conference on Bioinformatics and Biomedicine. Washington, DC, USA: IEEE, 2015. 990-993 [8] 李丹霞, 韦玉科. 基于自适应阈值的舌像分割方法. 计算机技术与发展, 2011, 21(9): 63-65 doi: 10.3969/j.issn.1673-629X.2011.09.016Li Dan-Xia, Wei Yu-Ke. Tongue image segmentation method based on adaptive thresholds. Computer Technology and Development, 2011, 21(9): 63-65 doi: 10.3969/j.issn.1673-629X.2011.09.016 [9] 蒋依吾, 陈建仲, 张恒鸿. 电脑化中医舌诊系统. 中国中西医结合杂志, 2000, 20(2): 145-147 doi: 10.3321/j.issn:1003-5370.2000.02.023Jiang Yi-Wu, Chen Jian-Zhong, Zhang Heng-Hong. Computerized Chinese medicine tongue diagnosis system. Chinese Journal of Integrated Traditional and Western Medicine, 2000, 20(2): 145-147 doi: 10.3321/j.issn:1003-5370.2000.02.023 [10] Zhi L, Yan J Q, Zhou T, Tang Q L. Tongue shape detection based on b-spline. In: Proceedings of the 2006 International Conference on Machine Learning and Cybernetics. Dalian, China: IEEE, 2006. 3829-3832 [11] Yu S Y, Yang J, Wang Y G, Zhang Y. Color active contour models based tongue segmentation in traditional Chinese medicine. In: Proceedings of the 1st International Conference on Bioinformatics and Biomedical Engineering. Wuhan, China: IEEE, 2007. 1065-1068 [12] Li X, Li J, Wang D. Automatic tongue image segmentation based on histogram projection and matting. In: Proceedings of the 2014 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, 2014: 76-81 [13] Fu Z C, Li W, Li X Q, Li F F, Wang Y Q. Automatic tongue location and segmentation. In: Proceedings of the 2008 International Conference on Audio, Language, and Image Processing. Shanghai, China: IEEE, 2008. 1050-1055 [14] Xie T, Xia C M, Chen F F, Zhang S L, Zhang Y. A method of tongue image segmentation based on kernel FCM. In: Proceedings of the 9th International Congress on Image and Signal Processing, BioMedical Engineering, and Informatics. Datong, China: IEEE, 2016. 319-324 [15] Liu Z, Yan J Q, Zhang D, Li Q L. Automated tongue segmentation in hyperspectral images for medicine. Applied Optics, 2007, 46(34): 8328-8334 doi: 10.1364/AO.46.008328 [16] Li X Q, Li J D, Wang D. Automatic tongue image segmentation based on histogram projection and matting. In: Proceedings of the 2014 IEEE International Conference on Bioinformatics and Biomedicine. Belfast, UK: IEEE, 2014. 76-81 [17] Xu W T, Kanawong R, Xu D, Li S, Ma T, Zhang G X, et al. An automatic tongue detection and segmentation framework for computer-aided tongue image analysis. In: Proceedings of the 13th IEEE International Conference on e-Health Networking, Applications and Services. Columbia, MO, USA: IEEE, 2011. 189-192 [18] Pang B, Zhang D, Wang K Q. The bi-elliptical deformable contour and its application to automated tongue segmentation in Chinese medicine. IEEE Transactions on Medical Imaging, 2005, 24(8): 946-956 doi: 10.1109/TMI.2005.850552 [19] Shi M J, Li G Z, Li F F, Xu C. Computerized tongue image segmentation via the double geo-vector flow. Chinese Medicine, 2014, 9: Article No. 7 doi: 10.1186/1749-8546-9-7 [20] Kass M, Witkin A, Terzopoulos D. Snakes: Active contour models. International Journal of Computer Vision, 1988, 1(4): 321-331 doi: 10.1007/BF00133570 [21] Kang D J. A fast and stable snake algorithm for medical images. Pattern Recognition Letters, 1999, 20(5): 507-512 doi: 10.1016/S0167-8655(99)00019-7 [22] Luo H, Gaborski R, Acharya R. Automatic segmentation of lung regions in chest radiographs: A model guided approach. In: Proceedings of the 2000 International Conference on Image Processing. Vancouver, BC, Canada: IEEE, 2000. 483-486 [23] McInerney T, Terzopoulos D. Deformable models in medical image analysis: A survey. Medical Image Analysis, 1996, 1(2): 91-108 doi: 10.1016/S1361-8415(96)80007-7 [24] 王爱民, 沈兰荪. 图像分割研究综述. 测控技术, 2000, 19(5): 1-6, 16 doi: 10.3969/j.issn.1000-8829.2000.05.001Wang Ai-Min, Shen Lan-Sun. Study surveys on image segmentation. Measurement and Control Technology, 2000, 19(5): 1-6, 16 doi: 10.3969/j.issn.1000-8829.2000.05.001 [25] 沈兰荪, 王爱民. 图像分割的分级性研究. 北京工业大学学报, 2000, 26(3): 33-38 doi: 10.3969/j.issn.0254-0037.2000.03.007Shen Lan-Sun, Wang Ai-Min. Hierarchy of image segmentation. Journal of Beijing Polytechnic University, 2000, 26(3): 33-38 doi: 10.3969/j.issn.0254-0037.2000.03.007 [26] McInerney T, Terzopoulos D. Topologically adaptable snakes. In: Proceedings of the 1995 IEEE International Conference on Computer Vision. Cambridge, MA, USA: IEEE, 1995. 840-845 [27] 王艳清, 卫保国, 蔡轶珩, 沈兰荪. 一种基于先验知识的自动舌体分割算法. 电子学报, 2004, 32(3): 489-491 doi: 10.3321/j.issn:0372-2112.2004.03.031Wang Yan-Qing, Wei Bao-Guo, Cai Yi-Heng, Shen Lan-Sun. A knowledge-based arithmetic for automatic tongue segmentation. Acta Electronica Sinica, 2004, 32(3): 489-491 doi: 10.3321/j.issn:0372-2112.2004.03.031 [28] 王郁中, 杨杰, 周越, 郑元杰, 王忆勤. 图像分割技术在中医舌诊客观化研究中的应用. 生物医学工程学杂志, 2005, 22(6): 1128- 1133 doi: 10.3321/j.issn:1001-5515.2005.06.011Wang Yu-Zhong, Yang Jie, Zhou Yue, Zheng Yuan-Jie, Wang Yi-Qin. Image segmentation in tongue characterization. Journal of Biomedical Engineering, 2005, 22(6): 1128- 1133 doi: 10.3321/j.issn:1001-5515.2005.06.011 [29] 吴佳, 张永红, 白净, 翁维良, 吴煜, 韩垚, 等. 基于分水岭变换和主动轮廓模型的舌体轮廓图像分割. 清华大学学报(自然科学版), 2008, 48(6): 1040-1043 doi: 10.3321/j.issn:1000-0054.2008.06.032Wu Jia, Zhang Yong-Hong, Bai Jing, Weng Wei-Liang, Wu Yu, Han Yao, et al. Tongue contour image extraction using a watershed transform and an active contour model. Journal of Tsinghua University (Science and Technology), 2008, 48(6): 1040-1043 doi: 10.3321/j.issn:1000-0054.2008.06.032 [30] Zhai X M, Lu H D, Zhang L Z. Application of image segmentation technique in tongue diagnosis. In: Proceedings of the 2009 International Forum on Information Technology and Applications. Chengdu, China: IEEE, 2009. 768-771 [31] 覃武星, 李斌, 岳小强. 一种基于初始化Snake轮廓线的混合舌图像分割算法. 中国科学技术大学学报, 2010, 40(8): 807-811 doi: 10.3969/j.issn.0253-2778.2010.08.007Qin Wu-Xing, Li Bin, Yue Xiao-Qiang. A hybrid tongue image segmentation algorithm based on initialization of snake contours. Journal of University of Science and Technology of China, 2010, 40(8): 807-811 doi: 10.3969/j.issn.0253-2778.2010.08.007 [32] Pang B, Wang K Q, Zhang S, Zhang F M. On automated tongue image segmentation in Chinese medicine. In: Proceedings of the 2002 Object Recognition Supported by User Interaction for Service Robots. Quebec City, Quebec, Canada: IEEE, 2002. 616-619 [33] Guo J W, Yang Y K, Wu Q W, Su J L, Ma F. Adaptive active contour model based automatic tongue image segmentation. In: Proceedings of the 9th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics. Datong, China: IEEE, 2016. 1386-1390 [34] 孙晓琳, 庞春颖. 一种改进snake模型的中医舌体分割方法. 长春理工大学学报(自然科学版), 2013, 36(5): 154-156 doi: 10.3969/j.issn.1672-9870.2013.05.043Sun Xiao-Lin, Pang Chun-Ying. An improved snake model method on tongue segmentation. Journal of Changchun University of Science and Technology (Natural Science Edition), 2013, 36(5): 154-156 doi: 10.3969/j.issn.1672-9870.2013.05.043 [35] 王明英, 张新峰, 卓力. 一种基于Snakes模型的中医舌象分割改进算法. 测控技术, 2011, 30(5): 32-35 doi: 10.3969/j.issn.1000-8829.2011.05.007Wang Ming-Ying, Zhang Xin-Feng, Zhuo Li. An improved snakes model for tongue image segmentation. Measurement and Control Technology, 2011, 30(5): 32-35 doi: 10.3969/j.issn.1000-8829.2011.05.007 [36] Zhang H Z, Zuo W M, Wang K Q, Zhang D. A snake-based approach to automated segmentation of tongue image using polar edge detector. International Journal of Imaging Systems and Technology, 2006, 16(4): 103-112 doi: 10.1002/ima.20075 [37] Salakhutdinov R, Mnih A, Hinton G. Restricted Boltzmann machines for collaborative filtering. In: Proceedings of the 24th International Conference on Machine Learning. Corvalis, Oregon, USA: ACM, 2007. 791-798 [38] Shelhamer E, Long J, Darrell T. Fully convolutional networks for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(4): 640 -651 doi: 10.1109/TPAMI.2016.2572683 [39] He K M, Gkioxari G, Dollár P, Girshick R. Mask R-CNN. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(2): 386-397 doi: 10.1109/TPAMI.2018.2844175 [40] Ren S Q, He K M, Girshick R, Sun J. Faster R-CNN: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1137-1149 doi: 10.1109/TPAMI.2016.2577031 [41] Badrinarayanan V, Kendall A, Cipolla R. SegNet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12): 2481-2495 [42] Qu P L, Zhang H, Zhuo L, Zhang J, Chen G Y. Automatic tongue image segmentation for traditional Chinese medicine using deep neural network. In: Proceedings of the 13th International Conference on Intelligent Computing. Liverpool, UK: Springer, 2017. 247-259 [43] Ronneberger O, Fischer P, Brox T. U-Net: Convolutional networks for biomedical image segmentation. In: Proceedings of the 18th International Conference on Medical Image Computing and Computer-Assisted Intervention. Munich, Germany: Springer, 2015. 234-241 [44] Chen L C, Zhu Y K, Papandreou G, Schroff F, Adam H. Encoder-decoder with atrous separable convolution for semantic image segmentation. In: Proceedings of the 15th European Conference on Computer Vision. Munich, Germany: Springer, 2018. 833-851 [45] Chen L C, Papandreou G, Kokkinos I, Murphy K, Yuille A L. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(4): 834-848 doi: 10.1109/TPAMI.2017.2699184 [46] Zhao H S, Shi J P, Qi X J, Wang X G, Jia J Y. Pyramid scene parsing network. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, HI, USA: IEEE, 2017. 6230-6239 [47] Chen L C, Papandreou G, Schroff F, Adam H. Rethinking atrous convolution for semantic image segmentation. arXiv: 1706.05587, 2017. [48] He K M, Zhang X Y, Ren S Q, Sun J. Deep residual learning for image recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016. 770-778 [49] He K M, Zhang X Y, Ren S Q, Sun J. Spatial pyramid pooling in deep convolutional networks for visual recognition. In: Proceedings of the 13th European Conference on Computer Vision. Zurich, Switzerland: Springer, 2014. 346-361 [50] Ioffe S, Szegedy C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: Proceedings of the 32nd International Conference on Machine Learning. Lille, France: JMLR. org, 2015. 448-456 [51] Zhang D, Zhang H Z, Zhang B. Tongue Image Analysis. Singapore: Springer, 2017. [52] Huang Q, Dom B. Quantitative methods of evaluating image segmentation. In: Proceedings of the 1995 International Conference on Image Processing. Washington, DC, USA: IEEE, 1995. 53-56 -