-
摘要: 人脸表情是人际交往的重要渠道，识别人脸表情可促进对人心理状态和情感的理解.不同于常规的人脸表情，微表情是一种特殊的面部微小动作，可以作为判断人主观情绪的重要依据，在公共安防和心理治疗领域有广泛的应用价值.由于微表情具有动作幅度小、持续时间短的特点，对微表情的人工识别需要专业的培训，且识别正确率较低.近年来不少研究人员开始利用计算机视觉技术自动识别微表情，极大地提高了微表情的应用可行性.本文综述人脸微表情识别的定义和研究现状，总结微表情识别中的一些关键技术，探讨潜在的问题和可能的研究方向.Abstract: Facial expression is an important channel in social interaction. Reading facial expression can improve understanding of psychological condition and emotional status. Different from normal expressions, microexpression is a special kind of subtle facial action. It serves as a vital clue for affective estimation, and has broad applications in public security and psychological treatment. Recognizing microexpression requires professional training for human due to its short duration and subtle movement. So far a low recognition accuracy has been reported in the literature. In recent years, researchers have been studying microexpression recognition based on computer vision, which can largely improve the feasibility of such recognition. In this article, we introduce problem definition and current research status of microexpression, survey several representative techniques in this topic, and discuss some underlying issues and potential research directions.1) 本文责任编委 赖剑煌
-
表 1 现有微表情数据集
Table 1 Existing datasets of microexpressions
数据集 帧率 #被试 #微表情 #非微表情 诱导方式 标注方法 SMIC 100 6 76 76 自发 情绪 SMIC2/HS 100 20 164 164 自发 情绪 SMIC2/VIS 25 10 71 71 自发 情绪 SMIC2/NIR 25 10 71 71 自发 情绪 CASME 60 35 195 / 自发 情绪/FACS CASMEⅡ 200 35 247 / 自发 情绪/FACS USF-HD 29.7 / 100 181 模仿 微/非微表情 Polikovsky 200 10 / / 模仿 FACS 
下载: 导出CSV
表 2 现有微表情识别方法的识别准确率 (%) 对比
Table 2 Recognition accuracy (%) of existing approaches on common datasets
方法 CASME CASMEⅡ LBP-TOP[23] 37.43(4类) 46.46(5类) STCLQP[36] 57.31(4类) 58.39(5类) LBP-SIP[37] 36.84(4类) 46.56(5类) DTCM[38] 64.95(4类) N/A TICS[43-44] 61.86(4类) 62.30(4类) STLBP-IP[45] N/A 59.51(5类) FDM[47] 56.14(5类) 45.93(5类) MDMO[48] 64.07(4类) 57.16(4类) DTSA[51] 46.90(5类) N/A MMPTR[54] N/A 80.2(4类) RPCA+LSDF[55] N/A 65.45(4类) Riesz小波[58] N/A 46.15(4类) EVM[59] N/A 67.21(4类)
下载: 导出CSV
表 3 现有微表情识别方法
Table 3 Existing approaches for microexpression recognition
方法 预处理方法 特征表达 学习算法 解决问题 文献[23] ASM、LWM、TIM LBP-TOP SVM、RF、MKL 检测/分类 文献[27, 70] ASM Strain tensor 阈值 检测 文献[28] ASM、面部分块 时空梯度 近邻投票 检测/分类 文献[36] ASM、TIM STCLQP SVM 检测/分类 文献[37] N/A LBP-SIP SVM 检测/分类 文献[38] AAM DTCM SVM、RF 检测/分类 文献[41] ASM Gabor GentlSVM 检测/分类 文献[42] N/A Gabor LDA、PCA、SVM 分类 文献[43-44] ASM TICS+LBP-TOP SVM 检测/分类 文献[45] N/A STLBP-IP SVM 检测/分类 文献[47] 光流场修正 FDM SVM 检测/分类 文献[48] DRMF、光流场对齐 MDMO SVM 检测/分类 文献[51] N/A DTSA变换 ELM 检测/分类 文献[53] ASM STCCA 最近邻 检测/分类 文献[54] N/A MMPTR变换 最近邻 检测/分类 文献[55] ASM、面部分块 RPCA+LSDF SVM 检测/分类 文献[57] 面部检测、分割 CBP-TOP ELM 分类 文献[58] N/A Riesz小波 MKL 分类 文献[59] ASM、LWM、TIM EVM处理后提前特征 SVM 分类 文献[62] N/A 特征点追踪 MKL 特定AU识别 文献[64] STASM、Procuste分析 几何形变特征 随机过程计算概率 检测 文献[66] 特征点定位 基于LBP的差异特征 阈值 检测 文献[67] DRMF 光流场 基于规则 阶段分割 文献[68] CLM CLM、LBP 基于规则 顶点定位 文献[71] N/A 时间平滑的Optical Strain SVM 检测/分类
下载: 导出CSV
-
[1] Shan C F, Gong S G, McOwan P W. Facial expression recognition based on local binary patterns: a comprehensive study. Image and Vision Computing, 2009, 27(6): 803-816 doi: 10.1016/j.imavis.2008.08.005 [2] Rahulamathavan Y, Phan R C W, Chambers J A, Parish D J. Facial expression recognition in the encrypted domain based on local fisher discriminant analysis. IEEE Transactions on Affective Computing, 2013, 4(1): 83-92 doi: 10.1109/T-AFFC.2012.33 [3] Wang S F, Liu Z L, Wang Z Y, Wu G B, Shen P J, He S, Wang X F. Analyses of a multimodal spontaneous facial expression database. IEEE Transactions on Affective Computing, 2013, 4(1): 34-46 doi: 10.1109/T-AFFC.2012.32 [4] 孙晓, 潘汀, 任福继.基于ROI-KNN卷积神经网络的面部表情识别.自动化学报, 2016, 42(6): 883-891 http://www.aas.net.cn/CN/abstract/abstract18879.shtmlSun Xiao, Pan Ting, Ren Fu-Ji. Facial expression recognition using ROI-KNN deep convolutional neural networks. Acta Automatica Sinica, 2016, 42(6): 883-891 http://www.aas.net.cn/CN/abstract/abstract18879.shtml [5] 刘帅师, 田彦涛, 王新竹.基于对称双线性模型的光照鲁棒性人脸表情识别.自动化学报, 2012, 38(12): 1933-1940 http://www.aas.net.cn/CN/abstract/abstract17855.shtmlLiu Shuai-Shi, Tian Yan-Tao, Wang Xin-Zhu. Illumination-robust facial expression recognition based on symmetric bilinear model. Acta Automatica Sinica, 2012, 38(12): 1933-1940 http://www.aas.net.cn/CN/abstract/abstract17855.shtml [6] 刘帅师, 田彦涛, 万川.基于Gabor多方向特征融合与分块直方图的人脸表情识别方法.自动化学报, 2011, 37(12): 1455-1463 http://www.aas.net.cn/CN/abstract/abstract17643.shtmlLiu Shuai-Shi, Tian Yan-Tao, Wan Chuan. Facial expression recognition method based on Gabor multi-orientation features fusion and block histogram. Acta Automatica Sinica, 2011, 37(12): 1455-1463 http://www.aas.net.cn/CN/abstract/abstract17643.shtml [7] Taheri S, Patel V M, Chellappa R. Component-based recognition of faces and facial expressions. IEEE Transactions on Affective Computing, 2013, 4(4): 360-371 doi: 10.1109/T-AFFC.2013.28 [8] El Mostafa M K A, Levine M D. Fully automated recognition of spontaneous facial expressions in videos using random forest classifiers. IEEE Transactions on Affective Computing, 2014, 5(2): 141-154 https://www.researchgate.net/publication/264387505_Fully_Automated_Recognition_of_Spontaneous_Facial_Expressions_in_Videos_Using_Random_Forest_Classifiers [9] Ekman P. Darwin, deception, and facial expression. Annals of the New York Academy of Sciences, 2003, 1000: 205-221 https://www.researchgate.net/publication/8882449_Darwin_Deception_and_Facial_Expression [10] Haggard E A, Isaacs K S. Micromomentary facial expressions as indicators of ego mechanisms in psychotherapy. Methods of Research in Psychotherapy. US: Springer, 1966. 154-165 [11] Ekman P, Friesen W. Nonverbal Leakage and Clues to Deception. Technical Report, DTIC Document, 1969. [12] Gottman J M, Levenson R W. A two-factor model for predicting when a couple will divorce: exploratory analyses using 14-year longitudinal data. Family Process, 2002, 41(1): 83-96 doi: 10.1111/famp.2002.41.issue-1 [13] Salter F, Grammer K, Rikowski A. Sex differences in negotiating with powerful males. Human Nature, 2005, 16(3): 306-321 doi: 10.1007/s12110-005-1013-4 [14] Whitehill J, Serpell Z, Lin Y C, Foster A, Movellan J R. The faces of engagement: automatic recognition of student engagementfrom facial expressions. IEEE Transactions on Affective Computing, 2014, 5(1): 86-98 doi: 10.1109/TAFFC.2014.2316163 [15] Pool L D, Qualter P. Improving emotional intelligence and emotional self-efficacy through a teaching intervention for university students. Learning and Individual Differences, 2012, 22(3): 306-312 doi: 10.1016/j.lindif.2012.01.010 [16] Porter S, ten Brinke L. Reading between the lies: identifying concealed and falsified emotions in universal facial expressions. Psychological Science, 2008, 19(5): 508-514 doi: 10.1111/j.1467-9280.2008.02116.x [17] Warren G, Schertler E, Bull P. Detecting deception from emotional and unemotional cues. Journal of Nonverbal Behavior, 2009, 33(1): 59-69 doi: 10.1007/s10919-008-0057-7 [18] Yan W J, Wu Q, Liang J, Chen Y H, Fu X L. How fast are the leaked facial expressions: the duration of micro-expressions. Journal of Nonverbal Behavior, 2013, 37(4): 217-230 https://www.researchgate.net/publication/245536570_How_Fast_Are_the_Leaked_Facial_Expressions_The_Duration_of_Micro-Expressions [19] Ekman P. MicroExpression Training Tool (METT). University of California, San Francisco, 2002. [20] Frank M G, Herbasz M, Sinuk K, Keller A, Nolan C. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In: Proceedings of the 2009 Annual Meeting of the International Communication Association. New York, 2009. http://www.allacademic.com/meta/p15018_index.htm [21] 吴奇, 申寻兵, 傅小兰.微表情研究及其应用.心理科学进展, 2010, 18(9): 1359-1368 http://www.cnki.com.cn/Article/CJFDTOTAL-XLXD201009002.htmWu Qi, Shen Xun-Bing, Fu Xiao-Lan. Micro-expression and its applications. Advances in Psychological Science, 2010, 18(9): 1359-1368 http://www.cnki.com.cn/Article/CJFDTOTAL-XLXD201009002.htm [22] 贲晛烨, 杨明强, 张鹏, 李娟.微表情自动识别综述.计算机辅助设计与图形学学报, 2014, 26(9): 1385-1395 http://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201409001.htmBen Xian-Ye, Yang Ming-Qiang, Zhang Peng, Li Juan. Survey on automatic micro expression recognition methods. Journal of Computer-Aided Design and Computer Graphics, 2014, 26(9): 1385-1395 http://www.cnki.com.cn/Article/CJFDTOTAL-JSJF201409001.htm [23] Pfister T, Li X B, Zhao G Y, Pietikäinen M. Recognising spontaneous facial micro-expressions. In: Proceedings of the 2011 IEEE International Conference on Computer Vision. Barcelona, Spain: IEEE, 2011. 1449-1456 [24] Li X B, Pfister T, Huang X H, Zhao G Y, Pietikäinen M. A spontaneous micro-expression database: inducement, collection and baseline. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai, China: IEEE, 2013. 1-6 [25] Yan W J, Wu Q, Liu Y J, Wang S J, Fu X L. CASME database: a dataset of spontaneous micro-expressions collected from neutralized faces. In: Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Shanghai, China: IEEE, 2013. 1-7 [26] Yan W J, Li X B, Wang S J, Zhao G Y, Liu Y J, Chen Y H, Fu X L. CASME Ⅱ: An improved spontaneous micro-expression database and the baseline evaluation. PLoS One, 2014, 9(1): e86041 http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.685.5991 [27] Shreve M, Godavarthy S, Goldgof D, Sarkar S. Macro-and micro-expression spotting in long videos using spatio-temporal strain. In: Proceedings of the 2011 IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Santa Barbara, CA, USA: IEEE, 2011. 51-56 [28] Polikovsky S, Kameda Y, Ohta Y. Facial micro-expression detection in hi-speed video based on facial action coding system (FACS). IEICE Transactions on Information and Systems, 2013, E96-D(1): 81-92 doi: 10.1587/transinf.E96.D.81 [29] Ekman P, Friesen W V. Facial Action Coding System. Palo Alto: Consulting Psychologists Press, 1977. [30] Cootes T F, Taylor C J, Cooper D H, Graham J. Active shape models-their training and application. Computer Vision and Image Understanding, 1995, 61(1): 38-59 doi: 10.1006/cviu.1995.1004 [31] Goshtasby A. Image registration by local approximation methods. Image and Vision Computing, 1998, 6(4): 255-261 [32] Zhou Z H, Zhao G Y, Pietikäinen M. Towards a practical lipreading system. In: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Colorado, USA: IEEE, 2011. 137-144 [33] Ojala T, Pietikäinen M, Maenpaa T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7): 971-987 doi: 10.1109/TPAMI.2002.1017623 [34] Zhao G Y, Pietikäinen M. Dynamic texture recognition using local binary patterns with an application to facial expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(6): 915-928 doi: 10.1109/TPAMI.2007.1110 [35] Huang X H, Zhao G Y, Hong X P, Pietikäinen M, Zheng W M. Texture description with completed local quantized patterns. Image Analysis. Berlin Heidelberg: Springer, 2013. 1-10 [36] Huang X H, Zhao G Y, Hong X P, Zheng W M, Pietikäinen M. Spontaneous facial micro-expression analysis using spatiotemporal completed local quantized patterns. Neurocomputing, 2016, 175: 564-578 doi: 10.1016/j.neucom.2015.10.096 [37] Wang Y D, See J, Phan P C W, Oh Y H. LBP with six intersection points: reducing redundant information in LBP-TOP for micro-expression recognition. In: Proceedings of the 12th Conference on Computer Vision, Singapore. Singapore: Springer, 2014. 21-23 [38] Lu Z Y, Luo Z Q, Zheng H C, Chen J K, Li W H. A delaunay-based temporal coding model for micro-expression recognition. Computer Vision-ACCV Workshops. Switzerland: Springer International Publishing, 2014. [39] Cootes T F, Edwards G J, Taylor C J. Active appearance models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001, 23(6): 681-685 doi: 10.1109/34.927467 [40] Barber B C, Dobkin D P, Huhdanpaa H. The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software, 1996, 22(4): 469-483 doi: 10.1145/235815.235821 [41] Wu W, Shen X B, Fu X L. The machine knows what you are hiding: an automatic micro-expression recognition system. In: Proceedings of the 4th International Conference on Affective Computing and Intelligent Interaction. Memphis, TN, USA: Springer-Verlag, 2011. 152-162 [42] Zhang P, Ben X Y, Yan R, Wu C, Guo C. Micro-expression recognition system. Optik——International Journal for Light and Electron Optics, 2016, 127(3): 1395-1400 https://www.researchgate.net/publication/283746871_Micro-Expression_Recognition_System [43] Wang S J, Yan W J, Li X B, Zhao G Y, Fu X L. Micro-expression recognition using dynamic textures on tensor independent color space. In: Proceedings of the 22nd International Conference on Pattern Recognition. Stockholm, Sweden: IEEE, 2014. 4678-4683 [44] Wang S J, Yan W J, Li X B, Zhao G Y, Zhou C G, Fu X L, Yang M H, Tao J H. Micro-expression recognition using color spaces. IEEE Transactions on Image Processing, 2015, 24(12): 6034-6047 https://www.ncbi.nlm.nih.gov/pubmed/26540689 [45] Huang X H, Wang S J, Zhao G Y, Piteikäinen M. Facial micro-expression recognition using spatiotemporal local binary pattern with integral projection. In: Proceedings of the 2015 IEEE International Conference on Computer Vision Workshops. Santiago, Chile: IEEE, 2015. 1-9 [46] Houam L, Hafiane A, Boukrouche A, Lespessailles E, Jennane R. One dimensional local binary pattern for bone texture characterization. Pattern Analysis and Applications, 2014, 17(1): 179-193 doi: 10.1007/s10044-012-0288-4 [47] Xu F, Zhang J P, Wang J Z. Microexpression identification and categorization using a facial dynamics map. IEEE Transactions on Affective Computing, PP(99): 1-1, DOI: 10.1109/TAFFC.2016.2518162 [48] Liu Y J, Zhang J K, Yan W J, Wang S J, Zhao G Y, Fu X L. A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Transactions on Affective Computing, 2016, 7(4): 299-310 doi: 10.1109/TAFFC.2015.2485205 [49] Asthana A, Zafeiriou S, Cheng S Y, Pantic M. Robust discriminative response map fitting with constrained local models. In: Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Portland, OR, USA: IEEE, 2013. 3444-3451 [50] Chaudhry R, Ravichandran A, Hager G, Vidal R. Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Miami, Florida: IEEE, 2009. 1932-1939 [51] Wang S J, Chen H L, Yan W J, Chen Y H, Fu X L. Face recognition and micro-expression recognition based on discriminant tensor subspace analysis plus extreme learning machine. Neural Processing Letters, 2014, 39(1): 25-43 https://www.researchgate.net/publication/236120483_Face_Recognition_and_Micro-expression_Recognition_Based_on_Discriminant_Tensor_Subspace_Analysis_Plus_Extreme_Learning_Machine [52] Huang G B, Zhu Q Y, Siew C K. Extreme learning machine: theory and applications. Neurocomputing, 2006, 70(1-3): 489-501 doi: 10.1016/j.neucom.2005.12.126 [53] Wang S J, Yan W J, Sun T K, Zhao G Y, Fu X L. Sparse tensor canonical correlation analysis for micro-expression recognition. Neurocomputing, 2016, 214: 218-232 doi: 10.1016/j.neucom.2016.05.083 [54] Ben X Y, Zhang P, Yan R, Yang M Q, Ge G D. Gait recognition and micro-expression recognition based on maximum margin projection with tensor representation. Neural Computing and Applications, 2015, 127(3): 1-18 https://www.researchgate.net/publication/283903772_Gait_recognition_and_micro-expression_recognition_based_on_maximum_margin_projection_with_tensor_representation [55] Wang S J, Yan W J, Zhao G Y, Fu X L, Zhou C G. Micro-expression recognition using robust principal component analysis and local spatiotemporal directional features. Computer Vision——ECCV 2014 Workshops. Switzerland: Springer International Publishing, 2014. [56] Fu X F, Wei W. Centralized binary patterns embedded with image euclidean distance for facial expression recognition. In: Proceedings of the 4th International Conference on Natural Computation. Jinan, China: IEEE, 2008. 115-119 [57] Guo Y C, Xue C H, Wang Y Z, Yu M. Micro-expression recognition based on CBP-TOP feature with ELM. Optik——International Journal for Light and Electron Optics, 2015, 126(23): 4446-4451 doi: 10.1016/j.ijleo.2015.08.167 [58] Oh Y H, Le Ngo A C, See J, Liong S T, Phan R C W, Ling H C. Monogenic riesz wavelet representation for micro-expression recognition. In: Proceedings of the 2015 IEEE International Conference on Digital Signal Processing. Singapore: IEEE, 2015. 1237-1241 [59] Li X B, Hong X P, Moilanen A, Huang X H, Pfister T, Zhao G Y, Pietikäinen M. Reading hidden emotions: spontaneous micro-expression spotting and recognition. arXiv Preprint arXiv: 1511.00423 [Online], available: https://arxiv.org/abs/1511.00423, February 20, 2017 [60] Wu H Y, Rubinstein M, Shih E, Guttag J, Durand F, Freeman W T. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics, 2012, 31(4): 65 https://www.researchgate.net/publication/254461914_Eulerian_Video_Magnification_for_Revealing_Subtle_Changes_in_the_World [61] Chavali G K, Bhavaraju S K N V, Adusumilli T, Puripanda V. Micro-expression Extraction for Lie Detection Using Eulerian Video (Motion and Color) Magnication [Master dissertation], Blekinge Institute of Technology, Swedish, 2014. [62] Yao S Q, He N, Zhang H Q, Yoshie O. Micro-expression recognition by feature points tracking. In: Proceedings of the 10th International Conference on Communications. Bucharest, Romania: IEEE, 2014. 1-4 [63] Kalal Z, Mikolajczyk K, Matas J. Tracking-learning-detec-tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(7): 1409-1422 doi: 10.1109/TPAMI.2011.239 [64] Xia Z Q, Feng X Y, Peng J Y, Peng X L, Zhao G Y. Spontaneous micro-expression spotting via geometric deformation modeling. Computer Vision and Image Understanding, 2016, 147: 87-94 doi: 10.1016/j.cviu.2015.12.006 [65] Milborrow S, Nicolls F. Active shape models with SIFT descriptors and MARS. In: Proceedings of the 2014 International Conference on Computer Vision Theory and Applications. Lisbon, Portugal: IEEE, 2014. 380-387 [66] Moilanen A, Zhao G Y, Pietikäinen M. Spotting rapid facial movements from videos using appearance-based feature difference analysis. In: Proceedings of the 2nd International Conference on Pattern Recognition. Stockholm, Sweden: IEEE, 2014. 1722-1727 [67] Patel D, Zhao G Y, Pietikäinen M. Spatiotemporal integration of optical flow vectors for micro-expression detection. Advanced Concepts for Intelligent Vision Systems. Switzerland: Springer International Publishing, 2015. 369-380 [68] Yan W J, Wang S J, Chen Y H, Zhao G Y, Fu X L. Quantifying micro-expressions with constraint local model and local binary pattern. Computer Vision——ECCV 2014 Workshops. Switzerland: Springer International Publishing, 2014. [69] Cristinacce D, Cootes T F. Feature detection and tracking with constrained local models. In: Proceedings of the 2006 BMVC. Edinburgh: BMVA, 2006. 929-938 [70] Shreve M, Godavarthy S, Manohar V, Goldgof D, Sarkar S. Towards macro-and micro-expression spotting in video using strain patterns. In: Proceedings of the 2009 IEEE Workshop on Applications of Computer Vision. Snowbird, UT, USA: IEEE, 2009. 1-6 [71] Liong S T, Phan R C W, See J, Oh Y H, Wong K. Optical strain based recognition of subtle emotions. In: Proceedings of the 2014 International Symposium on Intelligent Signal Processing and Communication Systems. Kuching, Sarawak, Malaysia: IEEE, 2014. 180-184 [72] House C, Meyer R. Preprocessing and descriptor features for facial micro-expression recognition [Online], available: https://web.stanford.edu/class/ee368/Project_Spring_1415/Reports/House_Meyer.pdf, February 20, 2017 -
图(7) / 表(3)
计量
- 文章访问数: 11123
- HTML全文浏览量: 2751
- PDF下载量: 4875
- 被引次数: 0
