2.793

2018影响因子

(CJCR)

  • 中文核心
  • EI
  • 中国科技核心
  • Scopus
  • CSCD
  • 英国科学文摘

留言板

尊敬的读者、作者、审稿人, 关于本刊的投稿、审稿、编辑和出版的任何问题, 您可以本页添加留言。我们将尽快给您答复。谢谢您的支持!

姓名
邮箱
手机号码
标题
留言内容
验证码

基于多层次特征融合的图像超分辨率重建

李金新 黄志勇 李文斌 周登文

李金新, 黄志勇, 李文斌, 周登文. 基于多层次特征融合的图像超分辨率重建. 自动化学报, 2021, x(x): 1−11 doi: 10.16383/j.aas.c200585
引用本文: 李金新, 黄志勇, 李文斌, 周登文. 基于多层次特征融合的图像超分辨率重建. 自动化学报, 2021, x(x): 1−11 doi: 10.16383/j.aas.c200585
Li Jin-Xin, Huang Zhi-Yong, Li Wen-Bin, Zhou Deng-Wen. Image super-resolution based on multi hierarchical features fusion network. Acta Automatica Sinica, 2021, x(x): 1−11 doi: 10.16383/j.aas.c200585
Citation: Li Jin-Xin, Huang Zhi-Yong, Li Wen-Bin, Zhou Deng-Wen. Image super-resolution based on multi hierarchical features fusion network. Acta Automatica Sinica, 2021, x(x): 1−11 doi: 10.16383/j.aas.c200585

基于多层次特征融合的图像超分辨率重建

doi: 10.16383/j.aas.c200585
详细信息
    作者简介:

    李金新:华北电力大学控制与计算机工程学院硕士研究生. 2018年获得河北建筑工程学院信息工程学院学士学位. 主要研究方向为计算机视觉和深度学习. E-mail: 1182227091@ncepu.edu.cn

    黄志勇:华北电力大学控制与计算机工程学院硕士研究生. 2018年获得华北电力大学控制与计算机工程学院学士学位. 主要研究方向为计算机视觉和深度学习. E-mail: 1182227193@ncepu.edu.cn

    李文斌:华北电力大学控制与计算机工程学院硕士研究生. 2017年获得上海电力学院计算机科学与技术学院学士学位. 主要研究方向为计算机视觉和深度学习. E-mail: 1182227108@ncepu.edu.cn

    周登文:华北电力大学控制与计算机工程学院教授. 长期从事图像处理方面的研究工作, 包括图像去噪、图像去马赛克、图像插值和图像超分辨率等. 当前的主要研究方向是神经网络和深度学习在图像处理和计算机视觉中的应用, 特别是, 图像超分辨率技术. 本文通信作者. E-mail: zdw@ncepu.edu.cn

Image Super-resolution Based on Multi Hierarchical Features Fusion Network

  • 摘要: 深度卷积神经网络显著改进了单图像超分辨率的性能. 更深的网络往往能获得更好的性能. 但是, 加深网络会导致参数量急剧增加, 限制了它在资源受限设备上的应用, 比如智能手机. 本文提出了一个融合多层次特征的轻量级单图像超分辨率网络. 网络构件主要是双层嵌套残差块. 为了更好地提取特征, 减少参数量, 每个残差块采用对称结构: 先两次扩张, 然后两次压缩通道数. 在残差块中, 通过添加自相关权重单元, 加权融合不同通道的特征信息. 实验证明: 我们的方法显著优于当前同类方法.
  • 图  1  (a)多层次特征融合网络结构图.(b)残差组结构图.(c)符号说明

    Fig.  1  (a)The architecture of multi hierarchical features fusion network (MHFN).(b)The structure of residual group.(c)Symbol description

    图  2  不同的残差块结构图

    Fig.  2  (a)The structure of different residual block

    图  3  自相关权重单元结构图

    Fig.  3  The structure of autocorrelation weight unit (ACW)

    图  4  标准测试集下×4视觉效果比较

    Fig.  4  Visual qualitative comparison on ×4 scale standard datasets

    图  5  标准测试集下×8视觉效果比较

    Fig.  5  Visual qualitative comparison on ×8 scale standard datasets

    表  1  残差组内不同数目双层嵌套残差块4倍因子, 200迭代周期下在Set5与DIV2K-10 数据集上的平均PSNR及参数量

    Table  1  Average PSNR and number of parameters of different DRB in residual group on Set5 and DIV2K-10 with scaling factor ×4 in 200 epochs

    数目 参数量 Set5 DIV2K-10
    5 1.23M 32.23 29.51
    6 1.47M 32.26 29.55
    7 1.71M 32.25 29.55
    下载: 导出CSV

    表  2  浅层特征映射单元支路不同卷积核设置的4倍因子模型, 200迭代周期下在Set5与DIV2K-10数据集上的平均PSNR

    Table  2  Average PSNR of different SFMU convolution kernel on Set5 and DIV2K-10 with scaling factor ×4 in 200 epochs

    卷积核设置 Set5 DIV2K-10
    32.22 29.52
    1 1 1 32.18 29.50
    3 3 3 32.24 29.53
    5 5 5 32.25 29.53
    1 3 5 32.26 29.55
    下载: 导出CSV

    表  3  不同结构的残差块对模型性能的影响, 记录模型在Set5与DIV2K-10数据集上4 倍因子200迭代周期下的平均PSNR

    Table  3  The influence of residual blocks with different structures on model performance. We report the average PSNR on Set5 and DIV2K-10 with scaling factor ×4 in 200 epochs

    残差块参数量 Set5 DIV2K-10
    I 73.8K 32.11 29.42
    II 53.5K 32.12 29.47
    下载: 导出CSV

    表  4  自相关权重单元对模型的影响, 记录模型在Set5 与DIV2K-10数据集上4倍因子200迭代周期下的平均PSNR

    Table  4  Effects of ACW. We report the average PSNR on Set5 and DIV2K-10 with scaling factor ×4 in 200 epochs

    ACW Set5 DIV2K-10
    × 32.11 29.42
    32.13 29.45
    下载: 导出CSV

    表  6  多路重建单元对重建效果的影响, 记录模型在Set5与DIV2K-10数据集上4倍因子200迭代周期下的平均PSNR

    Table  6  Effects of MPRU. We report the average PSNR on Set5 and DIV2K-10 with scaling factor ×4 in 200 epochs

    重建模块 参数量 Set5 DIV2K-10
    EDSR重建单元 297.16K 32.11 29.42
    MPRU 9.36K 32.13 29.47
    下载: 导出CSV

    表  5  各种SISR方法的平均PSNR值与SSIM值, 最好结果与次好结果分别用加粗和下划线标出

    Table  5  Average PSNR/SSIM of various SISR methods. Best and second best results are highlighted and underline

    放大倍数 模型 参数量 Set14 PSNR/SSIM B100 PSNR/SSIM Urban100 PSNR/SSIM Manga109 PSNR/SSIM
    ×2SRCNN 57K 32.42/0.9063 31.36/0.8879 29.50/0.8946 35.74/0.9661
    FSRCNN 12K 32.63/0.9088 31.53/0.8920 29.88/0.9020 36.67/0.9694
    VDSR 665K 33.03/0.9124 31.90/0.8960 30.76/0.9140 37.22/0.9729
    DRCN 1774K 33.04/0.9118 31.85/0.8942 30.75/0.9133 37.63/0.9723
    LapSRN 813K 33.08/0.9130 31.80/0.8950 30.41/0.9100 37.27/0.9740
    DRRN 297K 33.23/0.9136 32.05/0.8973 31.23/0.9188 37.92/0.9760
    MemNet 677K 33.28/0.9142 32.08/0.8978 31.31/0.9195 37.72/0.9740
    SRMDNF 1513K 33.32/0.9150 32.05/0.8980 31.33/0.9200 38.07/0.9761
    CARN 1592K 33.52/0.9166 32.09/0.8978 31.92/0.9256 38.36/0.9765
    MSRN 5930K 33.70/0.9186 32.23/0.9002 32.29/0.9303 38.69/0.9772
    SRFBN-S 282K 33.35/0.9156 32.00/0.8970 31.41/0.9207 38.06/0.9757
    CBPN 1036K 33.60/0.9171 32.17/0.8989 32.14/0.9279
    IMDN 694K 33.63/0.9177 32.19/0.8996 32.17/0.9283 38.88/0.9774
    MHFN(Ours) 1463K 33.79/0.9196 32.20/0.8998 32.40/0.9301 38.88/0.9774
    ×3SRCNN 57K 29.28/0.8209 28.41/0.7863 26.24/0.7989 30.59/0.9107
    FSRCNN 12K 29.43/0.8242 28.53/0.7910 26.43/0.8080 30.98/0.9212
    VDSR 665K 29.77/0.8314 28.82/0.7976 27.14/0.8279 32.01/0.9310
    DRCN 1774K 29.76/0.8311 28.80/0.7963 27.15/0.8276 32.31/0.9328
    DRRN 297K 29.96/0.8349 28.95/0.8004 27.53/0.8378 32.74/0.9390
    MemNet 677K 30.00/0.8350 28.96/0.8001 27.56/0.8376 32.51/0.9369
    SRMDNF 1530K 30.04/0.8370 28.97/0.8030 27.57/0.8400 33.00/0.9403
    CARN 1592K 30.29/0.8407 29.06/0.8034 27.38/0.8404 33.50/0.9440
    MSRN 6114K 30.41/0.8437 29.15/0.8064 28.33/0.8561 33.67/0.9456
    SRFBN-S 376K 30.10/0.8372 28.96/0.8010 27.66/0.8415 33.02/0.9404
    IMDN 703K 30.32/0.8417 29.09/0.8046 28.17/0.8519 33.61/0.9445
    MHFN(Ours) 1465K 30.40/0.8428 29.13/0.8056 28.35/0.8557 33.85/0.9460
    ×4SRCNN 57K 27.49/0.7503 26.90/0.7101 24.52/0.7221 27.66/0.8505
    FSRCNN 12K 27.59/0.7535 26.98/0.7150 24.62/0.7280 27.90/0.8517
    VDSR 665K 28.01/0.7674 27.29/0.7251 25.18/0.7524 28.83/0.8809
    DRCN 1774K 28.02/0.7670 27.23/0.7233 25.14/0.7510 28.98/0.8816
    LapSRN 813K 28.19/0.7720 27.32/0.7280 25.21/0.7560 29.09/0.8845
    DRRN 297K 28.21/0.7720 27.38/0.7284 25.44/0.7638 29.46/0.8960
    MemNet 677K 28.26/0.7723 27.40/0.7281 25.50/0.7630 29.42/0.8942
    SRMDNF 1555K 28.35/0.7770 27.49/0.7340 25.68/0.7730 30.09/0.9024
    CARN 1592K 28.60/0.7806 27.58/0.7349 26.07/0.7837 30.47/0.9084
    MSRN 6078K 28.63/0.7836 27.61/0.7380 26.22/0.7911 30.57/0.9103
    SRFBN-S 483K 28.45/0.7779 27.44/0.7313 25.71/0.7719 29.91/0.9008
    CBPN 1197K 28.63/0.7813 27.58/0.7356 26.14/0.7869
    IMDN 715K 28.58/0.7811 27.56/0.7353 26.04/0.7838 30.45/0.9075
    MHFN(Ours) 1468K 28.66/0.7830 27.61/0.7371 26.27/0.7909 30.74/0.9114
    ×8SRCNN 57K 23.86/0.5443 24.14/0.5043 21.29/0.5133 22.46/0.6606
    FSRCNN 12K 23.94/0.5482 24.21/0.5112 21.32/0.5090 22.39/0.6357
    VDSR 655K 23.20/0.5110 24.34/0.5169 21.48/0.5289 22.73/0.6688
    DRCN 1774K 24.25/0.5510 24.49/0.5168 21.71/0.5289 23.20/0.6686
    LapSRN 813K 24.45/0.5792 24.54/0.5293 21.81/0.5555 23.39/0.7068
    MSRN 6226K 24.88/0.5961 24.70/0.5410 22.37/0.5977 24.28/0.7517
    MHFN(Ours) 1490K 25.02/0.6426 24.80/0.5968 22.46/0.6170 24.60/0.7811
    下载: 导出CSV
  • [1] Shi W, Caballero J, Ledig C, Zhuang X, Bai W, Bhatia K, et al. Cardiac image super-resolution with global correspondence using multi-atlas patchmatch. In: Proceedings of Medical Image Computing and Computer-Assisted Intervention. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013. 916
    [2] Luo Y, Zhou L, Wang S, Wang Z. Video satellite imagery super resolution via convolutional neural networks. IEEE Geoscience and Remote Sensing Letters, 2017, 14(12): 2398−2402 doi: 10.1109/LGRS.2017.2766204
    [3] Zou W W W, Yuen P C. Very low resolution face recognition problem. IEEE Transactions on Image Processing, 2011, 21(1): 327−340
    [4] 孙旭, 李晓光, 李嘉锋, 卓力. 基于深度学习的图像超分辨率复原研 究进展. 自动化学报, 2017, 43(5): 697−709

    SUN Xu, LI Xiao-Guang, LI Jia-Feng, ZHUO Li. Review on deep learning based image super-resolution restoration algorithms. Acta Automatica Sinica, 2017, 43(5): 697−709
    [5] 周登文, 赵丽娟, 段然, 柴晓亮. 基于递归残差网络的图像超分辨率 重建. 自动化学报, 2019, 45(6): 1157−1165

    ZHOU Deng-Wen, ZHAO Li-Juan, DUAN Ran, CHAI XiaoLiang. Image super-resolution based on recursive residual networks. Acta Automatica Sinica, 2019, 45(6): 1157−1165
    [6] 张毅锋, 刘袁, 蒋程, 程旭. 用于超分辨率重建的深度网络递进学习 方法. 自动化学报, 2020, 46(2): 274−282

    ZHANG Yi-Feng, LIU Yuan, JIANG Cheng, CHENG Xu. A curriculum learning approach for single image super resolution. Acta Automatica Sinica, 2020, 46(2): 274−282
    [7] Dong C, Loy C C, He K, Tang X. Learning a deep convolutional network for image super-resolution. In: Proceedings of the European Conference on Computer Vision. Zurich, Switzerland: Springer, Cham, 2014. 184−199
    [8] LeCun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 1998, 86(11): 2278−2324 doi: 10.1109/5.726791
    [9] Kim J, Kwon Lee J, Mu Lee K. Accurate image superresolution using very deep convolutional networks. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016.1646−1654
    [10] He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016. 770−778
    [11] Kim J, Kwon Lee J, Mu Lee K. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE, 2016.1637−1645
    [12] Tai Y, Yang J, Liu X. Image super-resolution via deep recursive residual network. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017. 3147−3155
    [13] Li Z, Yang J, Liu Z, Yang X, Jeon G, Wu W. Feedback network for image super-resolution. In: Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE, 2019.3867−3876
    [14] Hui Z, Gao X, Yang Y, Wang X. Lightweight image superresolution with information multi-distillation network. In: Proceedings of the 27th ACM International Conference on Multimedia. New York, USA: Association for Computing Machinery, 2019. 2024−2032
    [15] Ahn N, Kang B, Sohn K A. Fast, accurate, and lightweight super-resolution with cascading residual network. In: Proceedings of the European Conference on Computer Vision. Zurich, Switzerland: Springer, Cham, 2018. 252−268
    [16] Zhu F, Zhao Q. Efficient single image super-resolution via hybrid residual feature learning with compact backprojection network. In: Proceedings of 2019 IEEE/CVF International Conference on Computer Vision Workshop. Seoul, Korea: IEEE, 2019. 2453−2460
    [17] Li J, Fang F, Mei K, Zhang G. Multi-scale residual network for image super-resolution. In: Proceedings of the European Conference on Computer Vision. Zurich, Switzerland: Springer, Cham, 2018. 517−532
    [18] Lai W S, Huang J B, Ahuja N, Yang M H. Deep laplacian pyramid networks for fast and accurate super-resolution. In: Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Honolulu, USA: IEEE, 2017. 5835−5843
    [19] Lim B, Son S, Kim H, Nah S, Lee K M. Enhanced deep residual networks for single image super-resolution. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, USA: IEEE, 2017. 1132−1140
    [20] Timofte R, Agustsson E, Gool L V, Yang M, Zhang L, Lim B, et al. Ntire 2017 challenge on single image superresolution: methods and results. In: Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Honolulu, USA: IEEE, 2017. 1110−1121
    [21] Zhang Y, Tian Y, Kong Y, Zhong B, Fu Y. Residual dense network for image super-resolution. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018.2472−2481
    [22] Liu J, Zhang W, Tang Y, Tang J, Wu G. Residual dense network for image super-resolution. In: Proceedings of 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020. 2356−2365
    [23] Zhang Y, Li K, Li K, Wang L, Zhong B, Fu Y. Image super-resolution using very deep residual channel attention networks. In: Proceedings of the European Conference on Computer Vision. Zurich, Switzerland: Springer, Cham, 2018. 286−301
    [24] Sandler M, Howard A, Zhu M, Zhmoginov A, Chen L. Mobilenetv2: inverted residuals and linear bottlenecks. In: Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 4510−4520
    [25] Szegedy C, Ioffe S, Vanhoucke V, Alemi A A. Inception-v4, inception-resNet and the impact of residual connections on learning. In: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence. San Francisco, USA: IEEE, 2017. 4278−4284
    [26] Wang Z, Chen J, Hoi S C H. Deep learning for image superresolution: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020
    [27] Kingma D P, Ba J. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014
    [28] Samlimans T, Kingma D P. Weight normalization: a simple reparameterization to accelerate training of deep neural networks. In: Proceedings of the 30th International Conference on Neural Information Processing Systems. Barcelona, Spain: IEEE, 2016. 901−909
    [29] Bevilacqua M, Roumy A, Guillemot C, Alberi-Morel M L. Low-complexity single-image super-resolution based on nonnegative neighbor embedding. In: Proceedings of the 23rd British Machine Vision Conference. Guildford, UK: BMVA Press, 2012. 135.1−135.10
    [30] Zeyde R, Elad M, Protter M. On single image scale-up using sparse-representations. In: Proceedings of International Conference on Curves and Surfaces. Berlin, Germany: Springer, Heidelberg, 2010. 71−1-730
    [31] Martin D, Fowlkes C, Tal D, Malik J. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In: Proceedings of the 2001 International Conference on Computer Vision. Vancouver, Canada: IEEE, 2015. 416−423
    [32] Huang J B, Singh A, Ahuja N. Single image super-resolution from transformed self-exemplars. In: Proceedings of the2015 IEEE Conference on Computer Vision and Pattern Recognition. Boston, USA: IEEE, 2015. 5197−5206
    [33] Matsui Y, Ito K, Aramaki Y, Fujimoto A, Ogawa T, Yamasaki T, et al. Sketch-based manga retrieval using manga109 dataset. Multimedia Tools and Applications, 2017, 76(20): 21811−21838 doi: 10.1007/s11042-016-4020-z
    [34] Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 2004, 13(4): 600−612 doi: 10.1109/TIP.2003.819861
    [35] Dong C, Loy C C, Tang X. Accelerating the super-resolution convolutional neural network. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Zurich, Switzerland: Springer, Cham, 2016.391−407
    [36] Tai Y, Yang J, Liu X, Xu C. Memnet: A persistent memory network for image restoration. In: Proceedings of the2017 IEEE International Conference on Computer Vision. Venice, Italy: IEEE, 2017. 4539−4547
    [37] Zhang K, Zuo W, Zhang L. Learning a single convolutional super-resolution network for multiple degradations. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE, 2018. 3262−3271
  • 加载中
计量
  • 文章访问数:  61
  • HTML全文浏览量:  27
  • 被引次数: 0
出版历程
  • 收稿日期:  2020-07-24
  • 录用日期:  2020-12-14
  • 网络出版日期:  2021-01-14

目录

    /

    返回文章
    返回