超分辨率图像重建方法综述

苏衡; 周杰; 张志浩

doi:10.3724/SP.J.1004.2013.01202

超分辨率图像重建方法综述

doi: 10.3724/SP.J.1004.2013.01202

苏衡^1,2,
周杰¹,
张志浩¹

1.
清华大学自动化系北京 100084;
2.
北京葫芦软件技术开发有限公司北京 100084

基金项目:

国家自然科学基金重大国际(地区)合作研究项目(61020106004);国家自然科学基金(61005023, 61021063);国家杰出青年科学基金项目(61225008);教育部博士点基金(20120002110033) 资助

详细信息

作者简介:
苏衡 2012 年获清华大学自动化系博士学位. 现就职于北京葫芦软件技术开发有限公司, 担任研究专员. 主要研究方向为超分辨率图像重建技术.E-mail: heng.su@hulu.com

计量
- 文章访问数: 5998
- HTML全文浏览量: 272
- PDF下载量: 7939
- 被引次数: 0
出版历程
- 收稿日期: 2011-08-31
- 修回日期: 2013-01-29
- 刊出日期: 2013-08-20

Survey of Super-resolution Image Reconstruction Methods

1.
Department of Automation, Tsinghua University, Beijing 100084;
2.
Beijing Hulu Inc., Beijing 100084

Funds:

Supported by Key International (Regional) Joint Research Pro- gram of National Natural Science Foundation of China (6102010 6004), National Natural Science Foundation of China (61005023, 61021063), National Science Fund for Distinguished Young Scholars (61225008), and Ph. D. Programs Foundation of Min- istry of Education of China (20120002110033)

摘要

摘要: 由于广泛的实用价值与理论价值,超分辨率图像重建(Super-resolution image reconstruction, SRIR 或 SR)技术成为计算机视觉与图像处理领域的一个研究热点, 引起了研究者的广泛关注. 本文将超分辨率图像重建问题按照不同的输入输出情况进行系统分类, 将超分辨率问题分为基于重建的超分辨率、视频超分辨率、单帧图像超分辨率三大类. 对于其中每一大类问题, 分别全面综述了该问题的发展历史、常用算法的分类及当前的最新研究成果等各种相关问题, 并对不同算法的特点进行了比较分析. 本文随后讨论了各不同类别超分辨率算法的互相融合和图像视频质量评价的方法, 最后给出了对这一领域未来发展的思考与展望.
- 超分辨率图像重建 /
- 计算机视觉 /
- 图像处理 /
- 方法综述
Abstract: Because of its extensive practical and theoretical values, the super-resolution image reconstruction (SRIR or SR) technique has become a hot topic in the areas of computer vision and image processing, attracting many researchers' attentions. This paper categorizes the SR problems according to their input and output conditions into three main categories: reconstruction-based SR, video SR and single image SR. For each category, the development history, common algorithm classes and state-of-the-art research achievements are reviewed comprehensively. We also analyze the characteristics of different algorithms. Afterwards, we discuss the combination of different super-resolution categories and the evaluation of image and video qualities. Thoughts and foresights of this field are given at the end of this paper.
- Super-resolution image reconstruction /
- computer vision /
- image processing /
- survey

HTML全文

参考文献(83)

[1]	Capel D, Zisserman A. Computer vision applied to super resolution. IEEE Signal Processing Magazine, 2003, 20(3): 75-86
[2]	Tsai R Y, Huang T S. Multiframe image restoration and registration. Advances in Computer Vision and Image Processing, 1984, 1: 317-339
[3]	Borman S, Stevenson R. Spatial Resolution Enhancement of Low-resolution Image Sequences: A Comprehensive Review with Directions for Future Research, Technical Report, Laboratory Image and Signal Analysis, University of Notre Dame, 1998
[4]	Chaudhuri S. Super-Resolution Imaging. Boston: Kluwer Academic Publishers, 2001
[5]	Park S C, Park M K, Kan M G. Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine, 2003, 20(3): 21-36
[6]	Van Ouwerkerk J D. Image super-resolution survey. Image and Vision Computing, 2006, 24(10): 1039-1052
[7]	Katartzis A, Petrou M. Current trends in super-resolution image reconstruction. Image Fusion: Algorithms and Applications. New York: Academic Press, 2008
[8]	Sun J, Zhu J J, Tappen M F. Context-constrained hallucination for image super-resolution. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Francisco, CA: IEEE, 2010. 231-238
[9]	Tai Y W, Liu S C, Brown M S, Lin S. Super resolution using edge prior and single image detail synthesis. In: Proceedings of the 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Francisco, CA: IEEE, 2010. 2400-2407
[10]	Farsiu S, Robinson M D, Elad M, Milanfar P. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 2004, 13(10): 1327-1344
[11]	Rhee S, Kang M. Discrete cosine transform based regularized high-resolution image reconstruction algorithm. Optical Engineering, 1999, 38(8): 1348-1356
[12]	Katsaggelos A K, Lay K T, Galatsanos N P. A general framework for frequency domain multi-channel signal processing. IEEE Transactions on Image Processing, 1993, 2(3): 417-420
[13]	Nguyen N, Milanfar P. An efficient wavelet-based algorithm for image superresolution. In: Proceedings of the 2000 International Conference on Image Processing. Vancouver, BC, Canada: IEEE, 2000, 2: 351-354
[14]	Ji H, Fermuller C. Robust wavelet-based super-resolution reconstruction: theory and algorithm. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2009, 31(4): 649-660
[15]	Lertrattanapanich S, Bose N K. High resolution image formation from low resolution frames using delaunay triangulation. IEEE Transactions on Image Processing, 2002, 11(12): 1427-1441
[16]	Sanchez-Beato A, Pajares G. Noniterative interpolation-based super-resolution minimizing aliasing in the reconstructed image. IEEE Transactions on Image Processing, 2008, 17(10): 1817-1826
[17]	Nasonov A V, Krylov A S. Fast super-resolution using weighted median filtering. In: Proceedings of the 20th International Conference on Pattern Recognition (ICPR). Istanbul: IEEE, 2010. 2230-2233
[18]	Lin S C, Chen C T. Reconstructing vehicle license plate image from low resolution images using nonuniform interpolation method. International Journal of Image Processing, 2007, 1(2): 21
[19]	Stark H, Oskoui P. High-resolution image recovery from image-plane arrays, using convex projections. Optical Society of America, Journal, A: Optics and Image Science, 1989, 6(11): 1715-1726
[20]	Banham M R, Katsaggelos A K. Digital image restoration. IEEE Signal Processing Magazine, 1997, 14(2): 24-41
[21]	Patti A J, Sezan M I, Murat T A. Superresolution video reconstruction with arbitrary sampling lattices and nonzero aperture time. IEEE Transactions on Image Processing, 1997, 6(8): 1064-1076
[22]	Kim J Y, Park R H, Yang S. Super-resolution using pocs-based reconstruction with artifact reduction constraints. In: Proceedings of the 2005 Visual Communications and Image Processing, 5960, 2005. 59605B
[23]	Patti A J, Altunbasak Y. Artifact reduction for set theoretic super resolution image reconstruction with edge adaptive constraints and higher-order interpolants. IEEE Transactions on Image Processing, 2001, 10(1): 179-186
[24]	Tom B, Katsaggelos A. Iterative algorithm for improving the resolution of video sequences. In: Proceedings of the 1996 SPIE, 2727, SPIE, 1996. 1430
[25]	Yu J, Xiao C B, Su K N. A method of gibbs artifact reduction for pocs super-resolution image reconstruction. In: Proceedings of the 8th International Conference on Signal Processing. Beijing, China: IEEE, 2006, 2: 1-4
[26]	Hennings-Yeomans P H, Baker S, Kumar B V K V. Simultaneous super-resolution and feature extraction for recognition of low-resolution faces. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[27]	Babacan S D, Molina R, Katsaggelos A K. Total variation super resolution using a variational approach. In: Proceedings of the 15th IEEE International Conference on Image Processing (ICIP). San Diego, CA: IEEE, 2008. 641-644
[28]	Chantas G, Galatsanos N, Likas A, Saunders M. Variational bayesian image restoration based on a product of t-distributions image prior. IEEE Transactions on Image Processing, 2008, 17(10): 1795-1805
[29]	Chantas G K, Galatsanos N P, Woods N A. Super-resolution based on fast registration and maximum a posteriori reconstruction. IEEE Transactions on Image Processing, 2007, 16(7): 1821-1830
[30]	Su H, Wu Y, Zhou J. Super-resolution without dense flow. IEEE Transactions on Image Processing, 2012, 21(4): 1782-1895
[31]	Bose N K, Lertrattanapanich S, Koo J. Advances in superresolution using L-curve. In: Proceedings of the 2001 IEEE International Symposium on Circuits and Systems. Sydney, NSW: IEEE, 2001, 2: 433-436
[32]	Yuan Q Q, Zhang L P, Shen H F, Li P X. Adaptive multiple-frame image super-resolution based on U-curve. IEEE Transactions on Image Processing, 2010, 19(12): 3157-3170
[33]	Huang Li-Li, Xiao Liang, Wei Zhi-Hui, Zhang Jun. A fast decoupling algorithm for image super-resolution reconstruction of space-invariant system. Acta Automatica Sinica, 2010, 36(2): 229-236(黄丽丽, 肖亮, 韦志辉, 张军. 空间移不变系统图像超分辨复原的快速解耦算法. 自动化学报, 2010, 36(2): 229-236)
[34]	Wang Q, Tang X O, Shum H. Patch based blind image super resolution. In: Proceedings of the 10th IEEE International Conference on Computer Vision (ICCV). Beijing, China: IEEE, 2005, 1: 709-716
[35]	Baboulaz L, Dragotti P L. Exact feature extraction using finite rate of innovation principles with an application to image super-resolution. IEEE Transactions on Image Processing, 2009, 18(2): 281-298
[36]	Sun Yan-Yue, He Xiao-Hai, Song Hai-Ying, Chen Wei-Long. A block-matching image registration algorithm for video super-resolution reconstruction. Acta Automatica Sinica, 2011, 37(1): 37-43(孙琰玥, 何小海, 宋海英, 陈为龙. 一种用于视频超分辨率重建的块匹配图像配准方法. 自动化学报, 2011, 37(1): 37-43)
[37]	Shen H F, Zhang L P, Huang B, Li P X. A map approach for joint motion estimation, segmentation, and super resolution. IEEE Transactions on Image Processing, 2007, 16(2): 479-490
[38]	He Y, Yap K H, Chen L, Chau L P. A nonlinear least square technique for simultaneous image registration and super-resolution. IEEE Transactions on Image Processing, 2007, 16(11): 2830-2841
[39]	Su H, Tang L, Wu Y, Tretter D, Zhou J. Spatially adaptive block-based super-resolution. IEEE Transactions on Image Processing, 2012, 21(3): 1031-1045
[40]	Takeda H, Farsiu S, Milanfar P. Kernel regression for image processing and reconstruction. IEEE Transactions on Image Processing, 2007, 16(2): 349-366
[41]	Takeda H, Milanfar P, Protter M, Elad M. Super-resolution without explicit subpixel motion estimation. IEEE Transactions on Image Processing, 2009, 18(9): 1958-1975
[42]	Protter M, Elad M. Super resolution with probabilistic motion estimation. IEEE Transactions on Image Processing, 2009, 18(8): 1899-1904
[43]	Protter M, Elad M, Takeda H, Milanfar P. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions on Image Processing, 2009, 18(1): 36-51
[44]	Gunturk B K, Altunbasak Y, Mersereau R M. Super-resolution reconstruction of compressed video using transform-domain statistics. IEEE Transactions on Image Processing, 2004, 13(1): 33-43
[45]	Krämer P, Hadar O, Benois-Pineau J, Domenger J P. Super-resolution mosaicing from mpeg compressed video. Signal Processing: Image Communication, 2007, 22(10): 845-865
[46]	Xu Z Q, Zhu X C. Super-resolution reconstruction of compressed video based on adaptive quantization constraint set. In: Proceedings of the 1st International Conference on Innovative Computing, Information and Control. Beijing, China: IEEE, 2006, 1: 281-284
[47]	Xu Z Q, Gan Z L, Zhu X C. Compressed video super-resolution reconstruction based on regularized algorithm. In: Proceedings of the 8th International Conference on Signal Processing. Beijing, China: IEEE, 2006, 2
[48]	Begin I, Ferrie F P. Comparison of super-resolution algorithms using image quality measures. In: Proceedings of the 3rd Canadian Conference on Computer and Robot Vision. Washington, DC, USA: IEEE Computer Society, 2006. 72
[49]	Mudenagudi U, Banerjee B, Kalra P K. Space-time super-resolution using graph-cut optimization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(5): 995-1008
[50]	Belekos S P, Galatsanos N P, Babacan S D, Katsaggelos A K. Maximum a posteriori super-resolution of compressed video using a new multichannel image prior. In: Proceedings of the 16th IEEE International Conference on Image Processing (ICIP). Cairo: IEEE, 2009. 2797-2800
[51]	Segall C A, Katsaggelos A K, Molina R, Mateos J. Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing, 2004, 13(7): 898-911
[52]	Su H, Wu Y, Zhou J. Adaptive incremental video super-resolution with temporal consistency. In: Proceedings of the 18th IEEE International Conference on Image Processing (ICIP). Brussels, Belgium: IEEE, 2011. 1149-1152
[53]	Kong D, Han M, Xu W, Tao H, Gong Y H. A conditional random field model for video super-resolution. In: Proceedings of the 18th International Conference on Pattern Recognition (ICPR). Hong Kong, China: IEEE, 2006, 3: 619-622
[54]	Zibetti M V W, Mayer J. Simultaneous super-resolution for video sequences. In: Proceedings of the 2005 IEEE International Conference on Image Processing (ICIP). Genova: IEEE, 2005, 1: I-877
[55]	Dai S Y, Han M, Xu W, Wu Y, Gong Y H. Soft edge smoothness prior for alpha channel super resolution. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Minneapolis, MN: IEEE, 2007. 1-8
[56]	Mallat S, Yu G S. Super-resolution with sparse mixing estimators. IEEE Transactions on Image Processing, 2010, 19(11): 2889-2900
[57]	Pentland A, Horowitz B. A practical approach to fractal-based image compression. In: Proceedings of the 1991 Data Compression Conference. Snowbird, UT: IEEE, 1991. 176- 185
[58]	Freeman W T, Pasztor E C. Learning low-level vision. In: Proceedings of the 7th IEEE International Conference on Computer Vision (ICCV). Kerkyra: IEEE, 1999, 2: 1182- 1189
[59]	Chang H, Yeung D Y, Xiong Y M. Super-resolution through neighbor embedding. In: Proceedings of the 2004 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington, DC, USA: IEEE, 2004, 1. I-275-I-282
[60]	Sun J, Zheng N N, Tao H, Shum H Y. Image hallucination with primal sketch priors. In: Proceedings of the 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR). Madison, WI, USA: IEEE, 2003, 2: II-729-36
[61]	Fan W, Yeung D Y. Image hallucination using neighbor embedding over visual primitive manifolds. In: Proceedings of the 2007 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Minneapolis, MN: IEEE, 2007. 1-7
[62]	Yang J C, Wright J, Huang T, Ma Y. Image super-resolution as sparse representation of raw image patches. In: Proceedings of the 2008 IEEE Conference of Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[63]	Adler A, Hel-Or Y, Elad M. A shrinkage learning approach for single image super-resolution with overcomplete representations. In: Proceedings of the 11th European Conference on Computer Vision (ECCV). 2010. Berlin, Heidelberg: Springer-Verlag, 622-635
[64]	Kim K I, Kwon Y. Single-image super-resolution using sparse regression and natural image prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(6): 1127-1133
[65]	Yang J C, Wright J, Huang T S, Ma Y. Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 2010, 19(11): 2861-2873
[66]	Wang J J, Zhu S H, Gong Y H. Resolution enhancement based on learning the sparse association of image patches. Pattern Recognition Letters, 2010, 31(1): 1-10
[67]	Wang J J, Zhu S H, Gong Y H. Resolution-invariant image representation and its applications. In: Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition CVPR 2009. Miami, FL: IEEE, 2009. 2512- 2519
[68]	Liu C, Shum H Y, Freeman W T. Face hallucination: theory and practice. International Journal of Computer Vision, 2007, 75(1): 115-134
[69]	Zhang W, Cham W K. Learning-based face hallucination in dct domain. In: Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Anchorage, AK: IEEE, 2008. 1-8
[70]	Hu Y, Lam K M, Qiu G P, Shen T Z. From local pixel structure to global image super-resolution: a new face hallucination framework. IEEE Transactions on Image Processing, 2011, 20(2): 433-445
[71]	Glasner D, Bagon S, Irani M. Super-resolution from a single image. In: Proceedings of the 2009 IEEE 12th International Conference on Computer Vision. Kyoto: IEEE, 2009. 349- 356
[72]	Zhao W, Sawhney H S. Is super-resolution with optical flow feasible? In: Proceedings of the 7th European Conference on Computer Vision. London, UK: Springer-Verlag 2002. 599-613
[73]	Costa G H, Bermudez J C M. Are registration errors always bad for super-resolution? In: Proceedings of the 2007 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Honolulu, HI: IEEE, 2007, 1: I-569- I-572
[74]	Baker S, Kanade T. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(9): 1167-1183
[75]	Lin Z C, Shum H Y. Fundamental limits of reconstruction-based superresolution algorithms under local translation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004, 26(1): 83-97
[76]	Tanaka M, Okutomi M. Theoretical analysis on reconstruc- tion-based super-resolution for an arbitrary PSF. In: Proceedings of the 2005 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). San Diego, CA, USA: IEEE, 2005, 2: 947-954
[77]	Robinson D, Milanfar P. Statistical performance analysis of super-resolution. IEEE Transactions on Image Processing, 2006, 15(6): 1413-1428
[78]	Wang Z, Bovik A C, Lu L G. Why is image quality assessment so difficult? In: Proceedings of the 2002 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). Orlando, FL, USA: IEEE, 2002, 4: IV-3313-IV-3316
[79]	Damera-Venkata N, Kite T D, Geisler W S, Evans B L, Bovik A C. Image quality assessment based on a degradation model. IEEE Transactions on Image Processing, 2000, 9(4): 636-650
[80]	Wang Z, Bovik A C, Sheikh H R, Simoncelli E P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, 13(4): 600-612
[81]	Sheikh H R, Bovik A C. Image information and visual quality. IEEE Transactions on Image Processing, 2006, 15(2): 430-444
[82]	Sheikh H R, Sabir M F, Bovik A C. A statistical evaluation of recent full reference image quality assessment algorithms. IEEE Transactions on Image Processing, 2006, 15(11): 3440-3451
[83]	Seshadrinathan K, Bovik A C. Motion tuned spatio-temporal quality assessment of natural videos. IEEE Transactions on Image Processing, 2010, 19(2): 335-350