强边缘提取网络用于非均匀运动模糊图像盲复原

黄彦宁; 李伟红; 崔金凯; 龚卫国

doi:10.16383/j.aas.c190654

强边缘提取网络用于非均匀运动模糊图像盲复原

doi: 10.16383/j.aas.c190654

黄彦宁^{1, 2,},
李伟红^{1, 2,},
崔金凯^{1, 2,},
龚卫国^{1, 2,}

1.
光电技术及系统教育部重点实验室重庆 400044
2.
重庆大学光电工程学院重庆 400044

基金项目: 国家科技惠民计划项目(2013GS500303), 广西科学研究与技术开发计划项目(桂科AA17129002)资助

详细信息

作者简介:
黄彦宁：重庆大学光电工程学院硕士研究生. 主要研究方向为图像处理技术. E-mail: 20122858@cqu.edu.cn

李伟红：重庆大学光电工程学院教授. 主要研究方向为图像处理技术, 模式识别. 本文通信作者.E-mail: weihongli@cqu.edu.cn

崔金凯：重庆大学光电工程学院博士研究生. 主要研究方向为图像处理技术. E-mail: jinkaicui@cqu.edu.cn

龚卫国：重庆大学光电工程学院教授. 主要研究方向为图像处理技术.E-mail: wggong@cqu.edu.cn

计量
- 文章访问数: 933
- HTML全文浏览量: 299
- PDF下载量: 196
- 被引次数: 0
出版历程
- 收稿日期: 2019-09-12
- 录用日期: 2020-05-07
- 网络出版日期: 2021-09-07
- 刊出日期: 2021-11-18

Strong Edge Extraction Network for Non-uniform Blind Motion Image Deblurring

HUANG Yan-Ning^{1, 2
,},
LI Wei-Hong^{1, 2
,},
CUI Jin-Kai^{1, 2
,},
GONG Wei-Guo^{1, 2
,}

1.
Key Laboratory of Optoelectronic Technology & Systems Ministry of Education, Chongqing 400044
2.
College of Optoelectronic Engineering, Chongqing University, Chongqing 400044

Funds: Supported by the National Science and Technolog Program for Public Wellbeing, China (2013GS500303), the Key Projects of Science and Technology Agency of Guangxi Province, China (Guike AA17129002)

More Information

Author Bio:
HUANG Yan-Ning　Master student at the College of Optoelectronic Engineering, Chongqing University. His main research interest is image processing technology

LI Wei-Hong　Professor at the College of Optoelectronic Engineering, Chongqing University. Her research interest covers image processing technology and pattern recognition. Corresponding author of this paper

CUI Jin-Kai　Ph. D. candidate at the College of Optoelectronic Engineering, Chongqing University. His main research interest is image processing technology

GONG Wei-Guo　Professor at the College of Optoelectronic Engineering, Chongqing University. His main research interest is image processing technology

摘要

摘要: 基于深度学习的非均匀运动图像去模糊方法已经获得了较好的效果. 然而, 现有的方法通常存在对边缘恢复不清晰的问题. 因此, 本文提出一种强边缘提取网络(Strong-edge extraction network, SEEN), 用于提取非均匀运动模糊图像的强边缘以提高图像边缘复原质量. 设计的强边缘提取网络由两个子网络SEEN-1和SEEN-2组成, SEEN-1实现双边滤波器的功能, 用于提取滤除了细节信息后的图像边缘. SEEN-2实现L₀平滑滤波器的功能, 用于提取模糊图像的强边缘. 本文还将对应网络层提取的强边缘特征图与模糊特征图叠加, 进一步利用强边缘特征. 最后, 本文在GoPro数据集上进行了验证实验, 结果表明: 本文提出的网络可以较好地提取非均匀运动模糊图像的强边缘, 复原图像在客观和主观上都可以达到较好的效果.
- 强边缘提取 /
- 梯度特征 /
- 卷积神经网络 /
- 非均匀运动模糊图像 /
- 模糊图像盲复原
Abstract: Although non-uniform motion image deblurring based on the deep learning has achieved better recovery effect, the most of the existing methods cannot recover the image edge well. In this paper, a strong edge extraction network (SEEN) is proposed for extracting the strong edges of the non-uniform motion blurry image to improve the quality of image deblurring. The designed SEEN is composed of two sub-networks, that is, SEEN-1 and SEEN-2. SEEN-1 is designed as a bilateral filter for extracting the edges of the image after filtering the image details. SEEN-2 is designed as an L₀ smoothing filter for extracting strong edges of the blurry image. Meanwhile, we also combine the strong edge features map and the blurry features map for further using the strong edge features. Finally, some experiments are executed on GoPro dataset and the results demonstrate that the proposed network can better extract the strong edge of the non-uniform motion blurry image, and obtain good results in both quality of visual perception and quantitative measurement.
- Strong edge extraction /
- gradient feature /
- convolutional neural network /
- non-uniform motion blurry image /
- blind image deblurring

HTML全文

图 1 均匀运动模糊图像和非均匀运动模糊图像及其模糊核示意图

Fig. 1 Uniform and non-uniform motion blurry image and their blur kernel diagram

下载: 全尺寸图片幻灯片

图 2 提出的非均匀运动模糊图像复原网络结构图

Fig. 2 Proposed network structure for the non-uniform motion blurry image deblurring

下载: 全尺寸图片幻灯片

图 3 模糊图像强边缘提取原理示意图

Fig. 3 Diagram of strong edge extraction of blurred image

下载: 全尺寸图片幻灯片

图 4 强边缘提取网络结构示意图

Fig. 4 Structure diagram of strong edge extraction network

下载: 全尺寸图片幻灯片

图 5 本文提出的双方向融合梯度计算方法示意图

Fig. 5 Diagram of gradient calculation method of two-directions fusion proposed in this paper

下载: 全尺寸图片幻灯片

图 6 改进残差模块

Fig. 6 Improvement of residual block

下载: 全尺寸图片幻灯片

图 7 不同大小滤波器的感受野

Fig. 7 Receptive field of different size filters

下载: 全尺寸图片幻灯片

图 8 数据集示例图像

Fig. 8 Sample images of dataset

下载: 全尺寸图片幻灯片

图 9 有无强边缘提取网络的非均匀模糊图像复原结果比较实例 ((a) 模糊图像; (b) 模糊图像梯度图;(c) SEEN-1 输出; (d) SEEN-2 输出 (强边缘); (e) 无强边缘提取网络的复原结果;(f) 有强边缘提取网络的复原结果; (g) 清晰图像)

Fig. 9 Comparison of restoration results of non-uniform motion blurry image with or without edge extraction network ((a) Blurry image; (b) Gradient of blurry image; (c) Output of SEEN-1; (d) Edge of SEEN-2 (Strong edge); (e) Restoration results of network without strong edge extraction network; (f) Restoration results of network with strong edge extraction network; (g) Clear image)

下载: 全尺寸图片幻灯片

图 10 强边缘提取网络中间结果分析

Fig. 10 The intermediate results analysis of strong edge extraction network

下载: 全尺寸图片幻灯片

图 11 强边缘提取网络效果分析

((i)模糊图像小块(30×30); (ii)模糊图像小块边缘(30×30); (iii) SEEN-1输出;(iv) SEEN-2输出; (v)灰度分布图; (vi)列灰度值加和曲线图)

Fig. 11 Effect analysis of strong edge extraction network

((i) Blurry image patch (30×30); (ii) Edge of blurry image patch (30×30); (iii) Output of SEEN-1; (iv) Output of SEEN-2; (v) Gray value distribution; (vi) Column gray value addition)

下载: 全尺寸图片幻灯片

12 对比实验结果

12 Results of comparative experiments

下载: 全尺寸图片幻灯片

图 13 对比实验图像的PSNR和SSIM指标值柱状图

Fig. 13 Histogram of PSNR and SSIM values of comparative experimental images

下载: 全尺寸图片幻灯片

表 1 交叉特征提取残差模块有效性验证实验结果(GoPro数据集)

Table 1 Validation experiment results of cross-resnet block (GoPro dataset)

评价指标	残差模块	交叉特征提取残差模块
PSNR	29.9800	30.2227
SSIM	0.8892	0.8944

下载: 导出CSV

表 2 对比实验图像的PSNR值比较

Table 2 PSNR value comparison of comparative experimental images

图像	文献 [4]	文献 [7]	文献 [10]	文献 [13]	文献 [21]	本文方法
(i)	26.3630	27.8248	29.8194	27.2473	27.2588	30.8864
(ii)	25.9098	25.3368	26.0540	22.5244	24.9011	27.0760
(iii)	22.5270	25.7279	29.4164	25.8877	26.7823	28.6511
(iv)	24.5868	27.4187	29.6548	25.7388	27.4837	30.0529

下载: 导出CSV

表 3 对比实验图像的SSIM值比较

Table 3 SSIM value comparison of comparative experimental images

图像	文献 [4]	文献 [7]	文献 [10]	文献 [13]	文献 [21]	本文方法
(i)	0.8031	0.8472	0.8821	0.7858	0.8290	0.8967
(ii)	0.7956	0.7593	0.8233	0.6034	0.7424	0.8335
(iii)	0.6502	0.7658	0.8738	0.7132	0.7928	0.8497
(iv)	0.7470	0.8170	0.8741	0.7125	0.8218	0.8764

下载: 导出CSV

表 4 复原图像的平均PSNR值和平均SSIM值(GoPro数据集)

Table 4 Average value of PSNR and SSIM of restored images (GoPro dataset)

评价指标	文献 [4]	文献 [7]	文献 [10]	文献 [13]	文献 [21]	文献 [22]	文献 [23]	本文方法
PSNR(dB)	24.5034	26.3980	30.1000	27.4648	24.5294	25.9200	29.5500	30.2227
SSIM	0.7621	0.8277	0.9137	0.8093	0.7761	0.7810	0.9340	0.8944

下载: 导出CSV

参考文献(23)

[1]	Xu L, Jia J Y. Two-phase kernel estimation for robust motion deblurring. In: Proceedings of the 2010 European Conference on Computer Vision. Crete, Greece: Springer, 2010. 157−170
[2]	Mesarovic V Z, Galatsanos N P. MAP and regularized constrained total least-squares image restoration. In: Proceedings of the 1st International Conference on Image Processing. Austin, TX, USA: IEEE, 1994. 177−181
[3]	Michael K N, Robert J P, Felipe P. A New Approach to Constrained Total Least Squares Image Restoration. Linear Algebra and its Applications, 2000, 316(1-3): 237-258 doi: 10.1016/S0024-3795(00)00115-4
[4]	Li W H, Chen Y Q, Chen R, Gong W G, Zhao B X. Hybrid order l0-regularized blur kernel estimation model for image blind deblurring. In: Proceedings of the 2017 International Symposium on Neural Networks. Sapporo, Japan: Springer, 2017. 239−247
[5]	Li W H, Chen R, Xu S W, Gong W G. Blind motion image deblurring using nonconvex higher-order total variation model. Journal of Electronic Imaging, 2016, 25(5): 053033.1-053033.19
[6]	Shen Z Y, Xu T F, Pan J S. Non-uniform Motion Deblurring with Kernel Grid Regularization. Signal Processing Image Communication, 2018, 62(0): 1-15
[7]	Chakrabarti A. A neural approach to blind motion deblurring. In: Proceedings of the 2016 European Conference on Computer Vision. Amsterdam, Netherlands: Springer, 2016. 221−235
[8]	Xu X Y, Pan J S, Zhang Y J, Yang M H. Motion Blur Kernel Estimation via Deep Learning. IEEE Transactions on Image Processing, 2018, 27(1): 194-205 doi: 10.1109/TIP.2017.2753658
[9]	Xu L, Ren J S, Liu C, Jia J Y. Deep convolutional neural network for image deconvolution. In: Proceedings of the 2014 International Conference on Neural Information Processing Systems. Montreal, Canada: Springer, 2014. 1790−1798
[10]	Tao X, Gao H Y, Shen X Y, Wang J, Jia J Y. Scale-Recurrent network for deep image deblurring. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, USA: IEEE, 2018. 8174−8182
[11]	Goodfellow I J, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D. Generative Adversarial Networks. Advances in Neural Information Processing Systems, 2014, (3): 2672-2680
[12]	吴梦婷, 李伟红, 龚卫国. 双框架卷积神经网络用于运动模糊图像盲复原. 计算机辅助设计与图形学学报, 2018, 30(12): 2327-2334 Wu M T, Li W H, Gong W G. Two-frame Convolutional Neural Network for Blind Motion Image Deblurring. Journal of Computer-Aided Design & Computer Graphics, 2018, 30(12): 2327-2334
[13]	Kupyn O, Budzan V, Mykhailych M, Mishkin D. DeblurGAN: Blind motion deblurring using conditional adversarial networks. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, USA: IEEE, 2018. 1−9
[14]	Tomasi C, Manduchi R. Bilateral filtering for gray and color images. In: Proceedings of the 1998 International Conference on Computer Vision. Bombay, India: IEEE, 1998. 839−846
[15]	Xu L, Lu C W, Xu Y, Jia J Y. Image Smoothing via L0 Gradient Minimization. ACM Transactions on Graphics, 2011, 30(6): 174.1-174.11
[16]	Wan R J, Shi B X, Duan L Y, Tan A H. CRRN: Multi-Scale guided concurrent reflection removal network. In: Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, Utah, USA: IEEE, 2018. 4777−4785
[17]	林景栋, 吴欣怡, 柴毅, 尹宏鹏. 卷积神经网络结构优化综述. 自动化学报, 2020, 46(1): 24-37 Lin J D, Wu X Y, Chai Y, Yin H P. Structure Optimization of Convolutional Neural Networks: A Survey. Acta Automatica Sinica, 2020, 46(1): 24-37
[18]	Deng J, Dong W, Socher R, Li L J, Li K, Li F F. ImageNet: A large-scale hierarchical image database. In: Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recogniltion. Florida, USA: IEEE, 2009: 1−8
[19]	Mao X D, Li Q, Xie H, Raymond Y K L, Wang Z, Stephen P S. Least squares generative adversarial networks. In: Proceedings of the 2017 International Conference on Computer Vision. Venice, Italy: IEEE, 2017: 2813−2821
[20]	PyTorch. PyTorch Documentation [Online], available: https://pytorch.org/docs/stable/index.html, September 1, 2019
[21]	Schuler C J, Hirsch M, Harmeling S, Scholkopf B. Learning to Deblur. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2016, 38(7): 1439-1451
[22]	孙季丰, 朱雅婷, 王恺. 基于DeblurGAN和低秩分解的去运动模糊. 华南理工大学学报(自然科学版), 2020, 48(1): 32-41 Sun J F, Zhu Y T, Wang K. Motion Deblurring Based on DeblurGAN and Low Rank Decomposition. Journal of South China University of Technology (Natural Science Edition), 2020, 48(1): 32-41
[23]	Kupyn O, Martyniuk T, Wu J R, Wang Z Y. DeblurGAN-v2: Deblurring (orders-of-magnitude) faster and better. In: Proceedings of the 2019 International Conference on Computer Vision. Seoul, Korea: IEEE, 2019. 8878−8887