2.793

2018影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于草图纹理和形状特征融合的草图识别

张兴园 黄雅平 邹琪 裴艳婷

张兴园, 黄雅平, 邹琪, 裴艳婷. 基于草图纹理和形状特征融合的草图识别. 自动化学报, 2020, 45(x): 1−10 doi: 10.16383/j.aas.c200070
引用本文: 张兴园, 黄雅平, 邹琪, 裴艳婷. 基于草图纹理和形状特征融合的草图识别. 自动化学报, 2020, 45(x): 1−10 doi: 10.16383/j.aas.c200070
Zhang Xing-Yuan, Huang Ya-Ping, Zou Qi, Pei Yan-Ting. Texture and Shape Feature Fusion Based Sketch Recognition. Acta Automatica Sinica, 2020, 45(x): 1−10 doi: 10.16383/j.aas.c200070
Citation: Zhang Xing-Yuan, Huang Ya-Ping, Zou Qi, Pei Yan-Ting. Texture and Shape Feature Fusion Based Sketch Recognition. Acta Automatica Sinica, 2020, 45(x): 1−10 doi: 10.16383/j.aas.c200070

基于草图纹理和形状特征融合的草图识别

doi: 10.16383/j.aas.c200070
基金项目: 中央高校基本科研业务费专项资金(2018YJS035, 2019JBZ104), 国家自然科学基金(61906013)资助
详细信息
    作者简介:

    张兴园:北京交通大学计算机与信息技术学院智能计算研究所博士研究生. 主要研究方向为深度学习, 数字图像处理和机器学习.E-mail: 15112071@bjtu.edu.cn

    黄雅平:北京交通大学计算机与信息技术学院智能计算研究所教授. 主要研究方向为机器学习与认知计算, 人工智能及应用和数字图像处理.Email: yphuang@bjtu.edu.cn

    邹琪:北京交通大学计算机与信息技术学院智能计算研究所教授. 主要研究方向为计算机视觉, 人工智能及应用和数字图像处理.Email: qzou@bjtu.edu.cn

    裴艳婷:北京交通大学计算机与信息技术学院智能计算研究所博士研究生. 主要研究方向为深度学习, 数字图像处理和机器学习.E-mail: 15112073@bjtu.edu.cn

Texture and Shape Feature Fusion Based Sketch Recognition

Funds: Supported by the Fundamental Research Funds for the Central Universities (2018YJS035, 2019JBZ104), National Natural Science Foundation of China (61906013)
  • 摘要: 人类具有很强的草图识别能力. 然而, 由于草图具有稀疏性和缺少细节的特点, 目前的深度学习模型在草图分类任务上仍然面临挑战. 目前的工作只是将草图看作灰度图像而忽略了不同草图类别间的形状表示差异. 本文提出一种端到端的手绘草图识别模型, 简称双模型融合网络(Dual-Model Fusion Network, DMF-Net), 它可以通过相互学习策略获取草图的纹理和形状信息. 具体来说, 该模型由两个分支组成: 一个分支能够从图像表示(即原始草图)中自动提取纹理特征, 另一个分支能够从图形表示(即基于点的草图)中自动提取形状特征. 此外, 提出视觉注意一致性损失来度量两个分支之间视觉显著图的一致性, 这样可以保证两个分支关注相同的判别性区域. 最终将分类损失、类别一致性损失和视觉注意一致性损失结合完成DMF-Net网络的优化. 本文在两个具有挑战性的数据集TU-Berlin数据集和Sketchy数据集上进行草图分类实验, 评估结果说明了DMF-Net显著优于基准方法并达到最佳性能.
  • 图  2  本文形状特征提取网络的原理框架示意图

    Fig.  2  Schematic diagram of Shape-net framework proposed in this paper

    图  1  本文算法总体框架图

    Fig.  1  The overall framework of our method

    图  3  TU-Berlin数据集上6个类别的受试者工作特征曲线及曲线下面积值

    Fig.  3  ROC Curves and AUC values of 6 classes in the TU-Berlin dataset

    图  4  TU-Berlin数据集上13个类别的分类准确率

    Fig.  4  Classification accuracy of 13 classes in the TU-Berlin dataset

    图  5  两种策略在草图点采样的结果示意图

    Fig.  5  The point sampling demonstration of two strategies on the sketch

    表  1  不同算法下在TU-Berlin数据集上分类准确率的比较

    Table  1  Comparison of sketch classification accuracy with different algorithms on the TU-Berlin dataset

    方法8406472
    Eitz (Knn hard)22%33%36%38%
    Eitz (Knn soft)26%39%43%44%
    Eitz (SVM hard)32%48%53%53%
    Eitz (SVM soft)33%50%55%55%
    FV size 1639%56%61%62%
    FV size 16 (SP)44%60%65%66%
    FV size 2441%60%64%65%
    FV size 24 (SP)43%62%67%68%
    SketchPoint50%68%71%74%
    AlexNet55%70%74%75%
    NIN51%70%75%75%
    VGGNet54%67%75%76%
    GoogLeNet52%69%76%77%
    Sketch-a-Net58%73%77%78%
    SketchNet58%74%77%80%
    Cousin Network59%75%78%80%
    Hybrid CNN57%75%80%81%
    LN58%76%82%82%
    SSDA59%76%82%84%
    DMF-Net60%77%85%86%
    下载: 导出CSV

    表  2  在TU-Berlin数据集和Sketchy数据集上实现草图分类的网络结构分析

    Table  2  Architecture design analysis for sketch classification on TU-Berlin and Sketchy

    方法TU-BerlinSketchy
    BN82.71%85.75%
    BN+GC83.93%86.49%
    BN+AC84.12%87.07%
    BN+CC84.75%87.36%
    BN+GC+CC85.47%87.64%
    BN+AC+CC85.51%87.71%
    BN+GC+AC+CC86.12%88.01%
    下载: 导出CSV

    表  3  利用双分支神经网络的草图分类准确率

    Table  3  Classification accuracy results using two-branch neural networks

    方法TU-BerlinSketchy
    纹理网络81.05%83.18%
    形状网络70.87%70.43%
    基础网络82.71%85.75%
    下载: 导出CSV

    表  4  不同层的分类准确率结果

    Table  4  Classification accuracy results using given feature levels

    方法TU-BerlinSketchy
    {4}85.83%87.23%
    {3,4}86.01%87.87%
    {2,3,4}86.06%87.93%
    {1,2,3,4}86.12%88.01%
    下载: 导出CSV

    表  5  两种采样策略在TU-Berlin数据集的分类准确率

    Table  5  Classification accuracy on TU-Berlin dataset using two sampling strategies

    方法分类准确率
    均匀采样86.12%
    随机采样86.09%
    下载: 导出CSV

    表  6  不同采样点数对分类准确率的影响

    Table  6  Effects of the point number for the classification accuracy

    点数数据集3264128256512
    TU-Berlin数据集81.87%82.75%83.23%84.34%85.42%
    Sketchy数据集83.37%84.35%84.83%85.90%87.36%
    点数数据集600750102412001300
    TU-Berlin数据集85.75%86.00%86.12%86.13%86.08%
    Sketchy数据集87.5%88.00%88.01%88.04%88.01%
    下载: 导出CSV
  • [1] Huang F, Canny J F, Nichols J. Swire: Sketch-based user interface retrieval. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. Glasgow, Scotland, UK: ACM, 2019. 1−10.
    [2] Dutta A, Akata Z. Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval. In: Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, LA, USA: IEEE, 2019. 5089−5098.
    [3] Riaz M, Yang Y, Song Y, Xiang T, Timothy M. Learning deep sketch abstraction. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake, USA: IEEE, 2018. 8014−8023.
    [4] Li K, Pang K, Song J, et al. Universal sketch perceptual grouping. In: Proceedings of the 15th IEEE European Conference on Computer Vision. Munich, Germany: IEEE, 2018. 582−597.
    [5] Chen W, Hays J. Sketchygan: Towards diverse and realistic sketch to image synthesis. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake, USA: IEEE, 2018. 9416−9425.
    [6] Zhang M, Zhang J, Chi Y, et al. Cross-domain face sketch synthesis. IEEE Access, 2019, 7: 98866−98874 doi: 10.1109/ACCESS.2019.2931012
    [7] Pang K, Li K, Yang Y, et al. Generalising fine-grained sketch-based image retrieval. In: Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE, 2019. 677−686.
    [8] 付晓, 沈远彤, 李宏伟, 程晓梅. 基于半监督编码生成对抗网络的图像分类模型. 自动化学报, 2020, 46(3): 531−539

    FU Xiao, SHEN Yuan-Tong, LI Hong-Wei, CHENG Xiao-Mei. A Semi-supervised Encoder Generative Adversarial Networks Model for Image Classification. ACTA AUTOMATICA SINICA, 2020, 46(3): 531−539
    [9] Zheng Y, Yao H, Sun X, et al. Sketch-specific data augmentation for freehand sketch recognition[J]. arXiv preprint arXiv: 1910.06038, 2019.
    [10] Hayat S, She K, Mateen M, Yu Y. Deep CNN-based Features for Hand-Drawn Sketch Recognition via Transfer Learning Approach. Editorial Preface From the Desk of Managing Editor, 2019, 10(9
    [11] Zhu M, Chen C, Wang N, Tang J, Bao W. Gradually focused fine-grained sketch-based image retrieval. PloS one, 2019, 14(5
    [12] Jabal M F A, Rahim M S M, Othman N Z S, Jupri Z. A comparative study on extraction and recognition method of cad data from cad drawings. In: Proceedings of the 2009 IEEE International Conference on Information Management and Engineering. Kuala Lumpur, Malaysia: IEEE, 2009. 709−713.
    [13] Eitz M, Hays J, Alexa M. How do humans sketch objects?. ACM Transactions on graphics (TOG), 2012, 31(4): 1−10
    [14] Schneider R G, Tuytelaars T. Sketch classification and classification-driven analysis using fisher vectors. ACM Transactions on Graphics (TOG), 2014, 33(6): 1−9
    [15] Dalal N, Triggs B. Histograms of oriented gradients for human detection. In: Proceedings of IEEE computer society conference on computer vision and pattern recognition. San Diego, California: IEEE, 2005. 886−893.
    [16] Lowe D G. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 2004, 60(2): 91−110 doi: 10.1023/B:VISI.0000029664.99615.94
    [17] Chang C, Lin C J. LIBSVM: A library for support vector machines. ACM transactions on intelligent systems and technology, 2011, 2(3): 1−27
    [18] 刘丽, 赵凌君, 郭承玉, 王亮, 汤俊. 图像纹理分类方法研究进展和展望. 自动化学报, 2018, 44(4): 584−607

    LIU Li, ZHAO Ling-Jun, GUO Cheng-Yu, WANG Liang, TANG Jun. Texture Classification: State-of-the-art Methods and Prospects. ACTA AUTOMATICA SINICA, 2018, 44(4): 584−607
    [19] Yu Q, Yang Y, Liu F, et al. Sketch-a-net: A deep neural network that beats humans. International journal of computer vision, 2017, 122(3): 411−425 doi: 10.1007/s11263-016-0932-3
    [20] He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Nevada, USA: IEEE, 2016. 770−778.
    [21] Bui T, Ribeiro L, Ponti M, et al. Sketching out the details: Sketch-based image retrieval using convolutional neural networks with multi-stage regression. Computers & Graphics, 2018, 71: 77−87
    [22] Dutta A, Akata Z. Semantically tied paired cycle consistency for zero-shot sketch-based image retrieval. In: Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, CA, USA: IEEE 2019. 5089−5098.
    [23] Li L, Zou C, Zheng Y, et al. Sketch-R2CNN: An Attentive Network for Vector Sketch Recognition[J]. arXiv preprint arXiv: 1811.08170, 2018.
    [24] Xu P, Song Z, Yin Q, Song Y, Wang L. Deep Self-Supervised Representation Learning for Free-Hand Sketch[J]. arXiv preprint arXiv: 2002.00867, 2020.
    [25] Xu P, Huang Y, Yuan T, et al. Sketchmate: Deep hashing for million-scale human sketch retrieval. In: Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake, USA, 2018. 8090−8098.
    [26] Jia Q, Yu M, Fan X, Li H. Sequential dual deep learning with shape and texture features for sketch recognition[J]. arXiv preprint arXiv: 1708.02716, 2017.
    [27] Liu Y, Tang K, Joneja A. Sketch-based free-form shape modelling with a fast and stable numerical engine. Comput. Graph, 2005: 771−786
    [28] 林景栋, 吴欣怡, 柴毅, 尹宏鹏. 卷积神经网络结构优化综述. 自动化学报, 2020, 46(1): 24−37

    LIN Jing-Dong, WU Xin-Yi, CHAI Yi, YIN Hong-Peng. Structure Optimization of Convolutional Neural Networks: A Survey. ACTA AUTOMATICA SINICA, 2020, 46(1): 24−37
    [29] Eldar Y, Lindenbaum M, Porat M, et al. The farthest point strategy for progressive image sampling. IEEE Transactions on Image Processing, 1997, 6(9): 1305−1315 doi: 10.1109/83.623193
    [30] Qi C, Su H, Mo K, Leonidas J. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA: IEEE, 2017. 77−85.
    [31] Hua B, Tran M, Yeung S. Pointwise Convolutional Neural Networks. In: Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake, USA: IEEE, 2018. 984−993.
    [32] Wang Y, Wu S, Huang H, Daniel C, Olga S. Patch-Based Progressive 3D Point Set Upsampling. In: Proceedings of 2019 IEEE Conference on Computer Vision and Pattern Recognition. Salt Lake, USA: IEEE, 2018. 5951−5960.
    [33] Li Y, Bu R, Sun M, et al. Pointcnn: Convolution on x-transformed points. In: Proceedings of the 2018 IEEE Advances in neural information processing systems. Montreal in Montreal, Canada: IEEE, 2018. 820−830.
    [34] Wang Y, Sun Y, Liu Z, et al. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019, 38(5): 1−12
    [35] Gold S, Rangarajan A, Lu C P, et al. New algorithms for 2D and 3D point matching: Pose estimation and correspondence. Pattern recognition, 1998, 31(8): 1019−1031 doi: 10.1016/S0031-3203(98)80010-1
    [36] Ho J, Yang M H, Rangarajan A, Baba C. A new affine registration algorithm for matching 2D point sets. In: Proceedings of 2007 IEEE Workshop on Applications of Computer Vision. Austin, Texas, USA: IEEE, 2007.
    [37] Shen Y, Feng C, Yang Y, Dong T. Mining point cloud local structures by kernel correlation and graph pooling. In: Proceedings of the IEEE conference on computer vision and pattern recognition. Salt Lake, USA: IEEE, 2018. 4548−4557.
    [38] Krizhevsky A, Sutskever I, Hinton G E. Imagenet classification with deep convolutional neural networks. In: Proceedings of the Advances in neural information processing systems. Doha, Qatar: IEEE, 2012. 1097−1105.
    [39] Sangkloy P, Burnell N, Ham C, James H. The sketchy database: learning to retrieve badly drawn bunnies. ACM Transactions on Graphics (TOG), 2016, 35(4): 1−12
    [40] Dey S, Riba P, Dutta A, Josep L. Doodle to search: Practical zero-shot sketch-based image retrieval. In: Proceedings of the 2019 IEEE Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019. 2179−2188.
    [41] Zhang H, Liu S, Zhang C, et al. Sketchnet: Sketch classification with web images. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, Nevada, USA: IEEE, 2016. 1105−1113.
    [42] Zhang K, Luo W, Ma L, Li H. Cousin network guided sketch recognition via latent attribute warehouse. In: Proceedings of the AAAI Conference on Artificial Intelligence. Hilton Hawaiian Village, Honolulu, Hawaii, USA: IEEE, 2019. 9203−9210.
    [43] Zhang H, She P, Liu Y, et al. Learning structural representations via dynamic object landmarks discovery for sketch recognition and retrieval. IEEE Transactions on Image Processing, 2019, 28(9): 4486−4499 doi: 10.1109/TIP.2019.2910398
    [44] Wang X, Chen X, Zha Z. Sketchpointnet: a compact network for robust sketch recognition. In: Proceedings of 2018 25th IEEE International Conference on Image Processing. Athens, Greece: IEEE, 2018. 2994−2998.
    [45] Lin M, Chen Q, Yan S. Network in network[J]. arXiv preprint arXiv: 1312.4400, 2013.
    [46] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv: 1409.1556, 2014.
    [47] Szegedy C, Liu W, Jia Y, et al. Going deeper with convolutions. In: Proceedings of the 2015 IEEE conference on computer vision and pattern recognition. Massachusetts, Boston, USA: IEEE, 2015. 1−9.
    [48] Zhang X, Huang Y, Zou Q, et al. A hybrid convolutional neural network for sketch recognition. Pattern Recognition Letters, 2019
    [49] Zheng Y, Yao H, Sun X, et al. Sketch-specific data augmentation for freehand sketch recognition[J]. arXiv preprint arXiv: 1910.06038, 2019.
    [50] Selvaraju R. et al. Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. In: Proceedings of the 2017 IEEE International Conference on Computer Vision. Las Vegas, Nevada, USA: IEEE, 2016. 618−626.
    [51] Dutt R, Sukhatme U. Mapping of shape invariant potentials under point canonical transformations. Journal of Physics A: Mathematical and General, 1992: L843
    [52] Mikolajczyk K and Cordelia S. An Affine Invariant Interest Point Detector. In: Proceedings of the 2002 IEEE European Conference on Computer Vision. Copenhagen, Denmark: IEEE, 2002.
    [53] Wang C, Samari B, Siddiqi K. Local spectral graph convolution for point set feature learning. In: Proceedings of the European conference on computer vision. Munich, Germany: IEEE, 2018. 52−66.
    [54] Simonovsky M, Nikos K. Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, Hawaii, USA: IEEE, 2017. 29−38.
    [55] Kipf T, Max W. Semi-Supervised Classification with Graph Convolutional Networks. In: Proceedings of the International Conference on Learning Representations. Toulon, France, 2017.
    [56] Hanley J, Barbara J. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 1982: 29−36
    [57] L ee, W, Paul D, Joseph N. Optimizing the Area Under a Receiver Operating Characteristic Curve With Application to Landmine Detection. IEEE Transactions on Geoscience and Remote Sensing, 2007: 389−397
  • 加载中
计量
  • 文章访问数:  25
  • HTML全文浏览量:  23
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-02-18
  • 录用日期:  2020-05-03

目录

    /

    返回文章
    返回