基于深度学习的图像超分辨率复原研究进展

孙旭; 李晓光; 李嘉锋; 卓力

doi:10.16383/j.aas.2017.c160629

基于深度学习的图像超分辨率复原研究进展

doi: 10.16383/j.aas.2017.c160629

北京工业大学信号与信息处理研究室北京 100124

基金项目:

北京市属高等学校人才强教计划 PHR(IHLB)

国家自然科学基金 61531006

北京市自然科学基金 4163071

国家自然科学基金 61471013

北京市教育委员会科技发展计划 KM201510005004

北京市自然科学基金 4142009

北京市教育委员会科技发展计划 KM201410005002

国家自然科学基金 61370189

北京市属高等学校高层次人才引进与培养计划 CIT & TCD201404043

北京市属高等学校高层次人才引进与培养计划 CIT & TCD20150311

国家自然科学基金 61372149

详细信息

作者简介:
孙旭北京工业大学计算机科学与技术专业硕士研究生.2015年获得内蒙古师范大学电子信息工程学士学位.主要研究方向为图像处理和模式识别.E-mail:993917172@emails.bjut.edu.cn

李嘉锋北京工业大学信号与信息处理实验室讲师.2009年于中国农业大学信息与电气工程学院获得学士学位, 并分别于2012年与2016年获得北京航空航天大学模式识别与智能系统专业硕士学位与博士学位.2014年至2015年赴美国匹兹堡大学做访问学者.主要研究方向为计算机视觉/图像增强, 图像复原.E-mail:lijiafeng@bjut.edu.cn

卓力北京工业大学教授.1992年于电子科技大学无线电技术系获工学学士学位, 1998年和2004年分别获得东南大学信号与信息处理专业硕士学位和北京工业大学模式识别与智能系统专业博士学位.主要研究方向为图像/视频编码和传输, 多媒体内容分析, 多媒体信息安全.E-mail:zhuoli@bjut.edu.cn

通讯作者:
李晓光北京工业大学副教授.2003年于北京工业大学电子与信息工程专业获得学士学位, 2008年获得北京工业大学博士学位.主要研究方向为计算机视觉/图像增强, 图像复原.E-mail:lxg@bjut.edu.cn

计量
- 文章访问数: 4010
- HTML全文浏览量: 2391
- PDF下载量: 5180
- 被引次数: 0
出版历程
- 收稿日期: 2016-09-06
- 录用日期: 2017-01-05
- 刊出日期: 2017-05-01

Review on Deep Learning Based Image Super-resolution Restoration Algorithms

Signal & Information Processing Laboratory, Beijing University of Technology, Beijing 100124

Funds:

Funding Project for Academic Human Resources Development in Institutions of Higher Learning under the Jurisdiction of Beijing Municipality PHR(IHLB)

Supported by National Natural Science Foundation of China 61531006

the Beijing Natural Science Foundation 4163071

Supported by National Natural Science Foundation of China 61471013

the Science and Technology Development Program of Beijing Education Committee KM201510005004

the Beijing Natural Science Foundation 4142009

the Science and Technology Development Program of Beijing Education Committee KM201410005002

Supported by National Natural Science Foundation of China 61370189

the Importation and Development of High-Caliber Talents Project of Beijing Municipal Institutions CIT & TCD201404043

the Importation and Development of High-Caliber Talents Project of Beijing Municipal Institutions CIT & TCD20150311

Supported by National Natural Science Foundation of China 61372149

More Information

Author Bio:
Master student in computer science and technology at Beijing University of Technology. He received his bachelar degree in electronic and information engineering from the Inner Mongolia Normal University in 2015. His research interest covers image processing and pattern recognition.

Assistant professor at Signal & Information Processing Laboratory, Beijing University of Technology. He received his bachelar degree from the College of Information and Electrical Engineering, China Agriculture University in 2009. He received his master degree and Ph. D. degree in pattern recognition and intelligence system from the Beihang University in 2012 and 2016, respectively. He was in the Department of Neurosurgery, University of Pittsburgh as a visiting scholar from 2014 to 2015. His research interest covers computer vision, image enhancement, and image restoration

Professor at Beijing University of Technology. She received her bachelor degree in radio technology from the University of Electronic Science and Technology in 1992, master degree in signal & information processing from the Southeast University in 1998, and Ph. D. degree in pattern recognition and intellectual system from Beijing University of Technology in 2004. Her research interest covers image/video coding and transmission, multimedia content analysis, and multimedia information security

Corresponding author: LI Xiao-Guang Associate professor at Beijing University of Technology. He received his bachelor degree in the electronic and information engineering, Beijing University of Technology in 2003. He received his Ph. D. degree from Beijing University of Technology in 2008. His research interest covers computer vision, image enhancement, and image restoration. Corresponding author of this paper

摘要

摘要: 图像超分辨率复原（Super resolution restoration，SR）技术是图像处理领域的研究热点，在视频监控、图像处理、刑侦分析等领域具有广泛的应用需求.近年来，深度学习在多媒体处理领域迅猛发展，基于深度学习的图像超分辨率复原技术已逐渐成为主流技术.本文主要对现有基于深度学习的图像超分辨率复原工作进行综述.从网络类型、网络结构、训练方法等方面分析现有技术的优势与不足，对其发展脉络进行梳理.在此基础上，本文进一步指出了基于深度学习的图像超分辨率复原技术的未来发展方向.
- 超分辨率复原 /
- 深度神经网络 /
- 卷积神经网络 /
- 循环神经网络
Abstract: Super resolution image restoration technology is a hot field of image processing in the field of video surveillance, image processing, forensic analysis, with a wide range of application requirements. In recent years, the rapid development of deep learning in the field of multimedia processing, deep learning based super-resolution images restoration has gradually become a mainstream technology. This paper reviews the existing deep learning based image super-resolution restoration work. In terms of network type, network structure, and training methods, the advantages and disadvantages of the prior art are analyzed and the development contexts are sorted out. On this basis, the paper further points out the future direction of the restoration technique based on deep learning of the super-resolution image.
- Super resolution restoration (SR) /
- deep neural networks /
- convolutional neural network (CNN) /
- recurrent neural network
注释:

1) 本文责任编委王亮

HTML全文

图 1 “Butterfly”、“Zebra”、“Comic”图像, 不同SR算法重建效果比较

Fig. 1 Comparison of reconstructed images with various SR methods under the "Butterfly", "Zebra", "Comic" images

下载: 全尺寸图片幻灯片

表 1 各类基于前馈深度网络的超分辨率算法比较表

Table 1 Comparison of different feed-forward deep network-based super-resolution algorithms

算法名称	网络结构	训练策略	算法目标	算法运行速度	生成图像质量
SRCNN^[30]	三层卷积	SGD	CNN与SC结合	一般	好
VDSR^[31]	VGG	梯度裁剪、残差学习	CNN网络加深	较快	较好
SRCNN-pr^[36]	三层卷积+特征提取	SGD、多任务学习	整合先验	稍好	好
SCN^[37]	LISTA	SGD、级联网络	学习稀疏先验	稍好	较好
CSCSR^[39]	三层卷积	ADMM	学习滤波器	较慢	稍好

下载: 导出CSV

表 2 各类基于反馈深度网络的超分辨率算法比较表

Table 2 Comparison of different feed-back deep network-based super-resolution algorithms

算法名称	网络结构	训练策略	算法目标	算法运行速度	生成图像质量
FD^[42]	反卷积网络	FISTA	加快训练速度	较快	一般
DRCN^[43]	递归网络	递归监督	层间信息连接	较慢	好
DEGREE^[44]	循环递归网络	损失函数先验	信息指导网络建设	一般	较好

下载: 导出CSV

表 3 各类基于双向深度网络的超分辨率算法比较表

Table 3 Comparison of different bi-directional deep network-based super-resolution algorithms

算法名称	网络结构	训练策略	算法目标	算法运行速度	生成图像质量
RBM^[45]	RBM	对比散度	加快训练速度	较快	一般
RBM^[46]	多个RBM堆叠	SGD	恢复图像细节信息	一般	较好
DNC^[47]	NLSS+CLA	BFGS	高频纹理增强	一般	好

下载: 导出CSV

表 4 Set5、Set14和BSD100数据集, 不同SR算法重建效果比较(PSNR)

Table 4 Comparison of reconstructed images with various SR methods (PSNR), on Set5, Set14, BSD100 benchmark data

数据集	放大倍数	ANR^[48]	A+^[49]	SRCNN^[30]	VDSR^[31]	DRCN^[43]	SCN^[37]	IA^[50]	JOR^[51]	DEGREE^[44]
Set5	×3	31.92	32.59	32.75	33.66	33.82	33.10	33.46	32.55	33.39
Set5	×4	29.69	30.28	30.48	31.35	31.53	30.86	31.10	30.19	31.03
Set14	×3	28.65	29.13	29.28	29.77	29.76	29.41	29.69	29.09	29.61
Set14	×4	26.85	27.32	27.49	28.01	28.02	27.64	27.88	27.26	27.73
BSD100	×3	27.89	28.29	28.29	28.82	28.80	28.50	28.76	28.17	28.63
BSD100	×4	26.51	26.82	26.84	27.29	27.23	27.03	27.25	26.74	27.07

下载: 导出CSV

参考文献(54)

[1]	卓力, 王素玉, 李晓光.图像/视频的超分辨率复原.北京:人民邮电出版社, 2011. 349 Zhuo Li, Wang Su-Yu, Li Xiao-Guang. Image/Video Super Resolution. Beijing: The People's Posts and Telecommunications Press, 2011. 349
[2]	张晓玲. 遥感图像的压缩和超分辨率复原技术研究[博士学位论文], 北京工业大学, 中国, 2006. Zhang Xiao-Ling. Research on compression and super resolution of remote sensing imagery [Ph.D. dissertation], Beijing University of Technology, China, 2006.
[3]	Baker S, Kanade T. Limits on super-resolution and how to break them. In: Proceedings of the 2000 IEEE Conference on Computer Vision and Pattern Recognition. Hilton Head Island, SC, USA: IEEE, 2000, 2: 372-379
[4]	Van Ouwerkerk J D. Image super-resolution survey. Image and Vision Computing, 2006, 24(10): 1039-1052 doi: 10.1016/j.imavis.2006.02.026
[5]	苏衡, 周杰, 张志浩.超分辨率图像重建方法综述.自动化学报, 2013, 39(8): 1202-1213 http://www.aas.net.cn/CN/abstract/abstract18151.shtml Su Heng, Zhou Jie, Zhang Zhi-Hao. Survey of super-resolution image reconstruction methods. Acta Automatica Sinica, 2013, 39(8): 1202-1213 http://www.aas.net.cn/CN/abstract/abstract18151.shtml
[6]	Harris J L. Diffraction and resolving power. Journal of the Optical Society of America, 1964, 54(7): 931-936 doi: 10.1364/JOSA.54.000931
[7]	Goodman J W. Introduction to Fourier Optics. San Francisco: McGraw-Hill, 1968.
[8]	Tsai R Y, Huang T S. Multiple frame image restoration and registration. In: Advances in Computer Vision and Image Processing. Greenwich, CT, England: JAI Press, 1984. 317-339
[9]	Aly H, Dubois E. Regularized image up-sampling using a new observation model and the level set method. In: Proceedings of the 2003 International Conference on Image Processing. Barcelona, Spain: IEEE, 2003, 2: Article No.Ⅲ-665-8
[10]	Aly H A, Dubois E. Image up-sampling using total-variation regularization with a new observation model. IEEE Transactions on Image Processing, 2005, 14(10): 1647-1659 doi: 10.1109/TIP.2005.851684
[11]	Zhang X L, Lam K M, Shen L S. Image magnification based on a blockwise adaptive Markov random field model. Image and Vision Computing, 2008, 26(9): 1277-1284 doi: 10.1016/j.imavis.2008.03.003
[12]	Chantas G K, Galatsanos N P, Woods N A. Super-resolution based on fast registration and maximum a posteriori reconstruction. IEEE Transactions on Image Processing, 2007, 16(7): 1821-1830 doi: 10.1109/TIP.2007.896664
[13]	Hardie R C, Barnard K J, Armstrong E E. Joint MAP registration and high-resolution image estimation using a sequence of undersampled images. IEEE Transactions on Image Processing, 1997, 6(12): 1621-1633 doi: 10.1109/83.650116
[14]	Protter M, Elad M. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 2009, 18(8): 1899-1904 doi: 10.1109/TIP.2009.2022440
[15]	Hu H, Kondi L P. A regularization framework for joint blur estimation and super-resolution of video sequences. In: Proceedings of the 2005 IEEE International Conference on Image Processing. Genoa, Italy: IEEE, 2005. Article No.Ⅲ-329-32
[16]	李晓光, 李风慧, 卓力.高分辨率与高动态范围图像联合重建研究进展.测控技术, 2012, 31(5): 8-12 http://www.cnki.com.cn/Article/CJFDTOTAL-IKJS201205004.htm Li Xiao-Guang, Li Feng-Hui, Zhuo Li. Research on High Resolution and High Dynamic Range Image Reconstruction. Measurement & Control Technology, 2012, 31(5): 8-12 http://www.cnki.com.cn/Article/CJFDTOTAL-IKJS201205004.htm
[17]	Baker S, Kanade T. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(9): 1167-1183 doi: 10.1109/TPAMI.2002.1033210
[18]	Freeman W T, Pasztor E C, Carmichael O T. Learning low-level vision. International Journal of Computer Vision, 2000, 40(1): 25-47 doi: 10.1023/A:1026501619075
[19]	Freeman W T, Jones T R, Pasztor E C. Example-based super-resolution. IEEE Computer Graphics and Applications, 2002, 22(2): 56-65 doi: 10.1109/38.988747
[20]	Li X G, Lam K M, Qiu G P, Shen L S, Wang S Y. Example-based image super-resolution with class-specific predictors. Journal of Visual Communication and Image Representation, 2009, 20(5): 312-322 doi: 10.1016/j.jvcir.2009.03.008
[21]	Su C Y, Zhuang Y T, Li H, Wu F. Steerable pyramid-based face hallucination. Pattern Recognition, 2005, 38(6): 813-824 doi: 10.1016/j.patcog.2004.11.007
[22]	Jiji C V, Chaudhuri S. Single-frame image super-resolution through contourlet learning. EURASIP Journal on Advances in Signal Processing, 2006, 2006: Article No.073767 https://www.researchgate.net/publication/27356019_Single-Frame_Image_Super-resolution_through_Contourlet_Learning/fulltext/0e605abaf0c46d4f0ab4581d/27356019_Single-Frame_Image_Super-resolution_through_Contourlet_Learning.pdf
[23]	Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE, 2004. 275-282
[24]	Chan T M, Zhang J P, Pu J, Huang H. Neighbor embedding based super-resolution algorithm through edge detection and feature selection. Pattern Recognition Letters, 2009, 30(5): 494-502 doi: 10.1016/j.patrec.2008.11.008
[25]	Yang J C, Wright J, Huang T, Ma Y. Image super-resolution as sparse representation of raw image patches. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Anchorage, Alaska, USA: IEEE, 2008. 1-8
[26]	Yang J C, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873 doi: 10.1109/TIP.2010.2050625
[27]	Yang C Y, Ma C, Yang M H. Single-image super-resolution: a benchmark. In: Proceedings of ECCV 2014 Conference on Computer Vision. Cham, Switzerland: Springer, 2014. 372-386
[28]	Schmidhuber J. Deep learning in neural networks: an overview. Neural Networks, 2015, 61: 85-117 doi: 10.1016/j.neunet.2014.09.003
[29]	胡传平, 钟雪霞, 梅林, 邵杰, 王建, 何莹.基于深度学习的图像超分辨率算法研究.铁道警察学院学报, 2016, 26(1): 5-10 http://www.cnki.com.cn/Article/CJFDTOTAL-TDBG201601001.htm Hu Chuan-Ping, Zhong Xue-Xia, Mei Lin, Shao Jie, Wang Jian, He Ying. The research on super-resolution method using deep learning. Journal of Railway Police College, 2016, 26(1): 5-10 http://www.cnki.com.cn/Article/CJFDTOTAL-TDBG201601001.htm
[30]	Dong C, Loy C C, He K M, Tang X O. Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2): 295-307 doi: 10.1109/TPAMI.2015.2439281
[31]	Kim J, Lee J K, Lee K M. Accurate image super-resolution using very deep convolutional networks. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016. 1646-1654
[32]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv: 1409.1556, 2015.
[33]	Pascanu R, Mikolov T, Bengio Y. On the difficulty of training recurrent neural networks. In: Proceedings of the 30th International Conference on Machine Learning. Atlanta, GA, USA: International Machine Learning Society, 2013. 2347-2355
[34]	潘宗序, 禹晶, 肖创柏, 孙卫东.基于多尺度非局部约束的单幅图像超分辨率算法.自动化学报, 2014, 40(10): 2233-2244 http://www.aas.net.cn/CN/abstract/abstract18498.shtml Pan Zong-Xu, Yu Jing, Xiao Chuang-Bai, Sun Wei-Dong. Single-image super-resolution algorithm based on multi-scale nonlocal regularization. Acta Automatica Sinica, 2014, 40(10): 2233-2244 http://www.aas.net.cn/CN/abstract/abstract18498.shtml
[35]	He K M, Zhang X Y, Ren S Q, Sun J. Spatial pyramid pooling in deep convolutional networks for visual recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(9): 1904-1916 doi: 10.1109/TPAMI.2015.2389824
[36]	Liang Y, Wang J J, Zhou S P, Gong Y H, Zheng N N. Incorporating image priors with deep convolutional neural networks for image super-resolution. Neurocomputing, 2016, 194: 340-347 doi: 10.1016/j.neucom.2016.02.046
[37]	Wang Z W, Liu D, Yang J C, Han W, Huang T. Deep networks for image super-resolution with sparse prior. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015. 370-378
[38]	Gregor K, Lecun Y. Learning fast approximations of sparse coding. In: Proceedings of the 27th International Conference on Machine Learning. Haifa, Israel: International Machine Learning Society, 2010. 399-406
[39]	Gu S H, Zuo W M, Xie Q, Meng D Y, Feng X C, Zhang L. Convolutional sparse coding for image super-resolution. In: Proceedings of the 2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015. 1823-1831
[40]	Zhong L W, Kwok J T. Fast stochastic alternating direction method of multipliers. In: Proceedings of the 31st International Conference on Machine Learning. Beijing, China: International Machine Learning Society, 2014. 78-86
[41]	Tian J, Ma K K. A survey on super-resolution imaging. Signal, Image and Video Processing, 2011, 5(3): 329-342 doi: 10.1007/s11760-010-0204-6
[42]	Krishnan D, Fergus R. Fast image deconvolution using hyper-laplacian priors. In: Proceedings of the 23rd Annual Conference on Neural Information Processing Systems. Vancouver, BC, Canada: NIPS, 2009. 1033-1041
[43]	Kim J, Lee J K, Lee K M. Deeply-recursive convolutional network for image super-resolution. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016. 1637-1645
[44]	Yang W H, Feng J S, Yang J C, Zhao F, Liu J Y, Guo Z M, Yan S C. Deep edge guided recurrent residual learning for image super-resolution. ArXiv: 1604.08671, 2016.
[45]	Gao J B, Guo Y, Yin M. Restricted Boltzmann machine approach to couple dictionary training for image super-resolution. In: Proceedings of the 20th IEEE International Conference on Image Processing (ICIP). Melbourne, VIC, Australia: IEEE, 2013. 499-503
[46]	Nakashika T, Takiguchi T, Ariki Y. High-frequency restoration using deep belief nets for super-resolution. In: Proceedings of the 9th International Conference on Signal-Image Technology & Internet-Based Systems. Kyoto, Japan: IEEE, 2013. 38-42
[47]	Cui Z, Chang H, Shan S G, Zhong B N, Chen X L. Deep network cascade for image super-resolution. In: Proceedings of the 13th European Conference on Computer Vision-ECCV 2014. Cham, Switzerland: Springer, 2014. 49-64
[48]	Timofte R, De V, Van Gool L. Anchored neighborhood regression for fast example-based super-resolution. In: Proceedings of the 14th IEEE International Conference on Computer Vision (ICCV). Sydney, NSW, Australia: IEEE, 2013. 1920-1927
[49]	Timofte R, De Smet V, Van Gool L. A+: adjusted anchored neighborhood regression for fast super-resolution. In: Proceedings of the 12th Asian Conference on Computer Vision. Cham, Switzerland: Springer, 2015. 111-126
[50]	Timofte R, Rothe R, Van Gool L. Seven ways to improve example-based single image super resolution. In: Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, NV, USA: IEEE, 2016. 1865-1873
[51]	Dai D, Timofte R, Van Gool L. Jointly optimized regressors for image super-resolution. Computer Graphics Forum, 2015, 34(2): 95-104 doi: 10.1111/cgf.2015.34.issue-2
[52]	徐冉, 张俊格, 黄凯奇.利用双通道卷积神经网络的图像超分辨率算法.中国图象图形学报, 2016, 21(5): 556-564 doi: 10.11834/jig.20160503 Xu Ran, Zhang Jun-Ge, Huang Kai-Qi. Image super-resolution using two-channel convolutional neural networks. Journal of Image and Graphics, 2016, 21(5): 556-564 doi: 10.11834/jig.20160503
[53]	刘娜, 李翠华.基于多层卷积神经网络学习的单帧图像超分辨率重建方法.中国科技论文, 2015, 10(2): 201-206 http://www.cnki.com.cn/Article/CJFDTOTAL-ZKZX201502017.htm Liu Na, Li Cui-Hua. Single image super-resolution reconstruction via deep convolutional neural network. China Sciencepaper, 2015, 10(2): 201-206 http://www.cnki.com.cn/Article/CJFDTOTAL-ZKZX201502017.htm
[54]	Nazzal M, Ozkaramanli H. Wavelet domain dictionary learning-based single image superresolution. Signal, Image and Video Processing, 2015, 9(7): 1491-1501 doi: 10.1007/s11760-013-0602-7