多视点稀疏测量的图像绘制方法

兰诚栋; 林宇鹏; 方大锐; 陈建

doi:10.16383/j.aas.c180199

多视点稀疏测量的图像绘制方法

doi: 10.16383/j.aas.c180199

兰诚栋^1, ,,
林宇鹏^1,,
方大锐^1,,
陈建^1,

1.
福州大学物理与信息工程学院福州 350116

基金项目:

国家自然科学基金 62001117

国家自然科学基金 61671152

国家自然科学基金 61471124

福建省自然科学基金 2017J01757

福建省自然科学基金 2014J01234

福州大学科研启动基金 GXRC-17034

详细信息

作者简介:
林宇鹏  福州大学物理与信息工程学院硕士研究生. 主要研究方向为图像处理. E-mail: 18850766818@163.com

方大锐  福州大学物理与信息工程学院硕士研究生. 主要研究方向为图像处理与压缩感知应用. E-mail: N161120076@fzu.edu.cn

陈建  福州大学物理与信息工程学院讲师. 主要研究方向为视频编码, 压缩感知, 点云压缩. E-mail: chenjian-fzu@163.com

通讯作者:
兰诚栋福州大学物理与信息工程学院副教授. 主要研究方向为图像处理与视频编码. 本文通信作者. E-mail: lancd@fzu.edu.cn

计量
- 文章访问数: 844
- HTML全文浏览量: 138
- PDF下载量: 124
- 被引次数: 0
出版历程
- 收稿日期: 2018-04-09
- 录用日期: 2018-10-11
- 刊出日期: 2021-04-23

Multi-view Sparse Measurement for Image-based Rendering Method

1.
College of Physics and Information Engineering, Fuzhou University, Fuzhou 350116

Funds:

National Natural Science Foundation of China 62001117

National Natural Science Foundation of China 61671152

National Natural Science Foundation of China 61471124

Natural Science Foundation of FuJian Province 2017J01757

Natural Science Foundation of FuJian Province 2014J01234

Fuzhou University Research Startup Fund GXRC-17034

More Information

Author Bio:
LIN Yu-Peng  Master student at the College of Physics and Information Engineering, Fuzhou University. His main research interest is image processing

FANG Da-Rui  Master student at the College of Physics and Information Engineering, Fuzhou University. His research interest covers image processing and the application of compressed sensing

CHEN Jian  Lecturer at the College of Physics and Information Engineering, Fuzhou University. Her research interest covers video coding, compressed sensing, and point cloud compression

Corresponding author: LAN Cheng-Dong Associate professor at the College of Physics and Information Engineering, Fuzhou University. His research interest covers image processing and video coding. Corresponding author of this paper

摘要

摘要: 为了减少所需采集的视频数据量, 基于图像绘制(Image-based rendering, IBR) 的前沿方法将稠密视点信息映射成压缩感知框架中的原始信号, 并将稀疏视点图像作为随机测量值, 但低维测量信号由所有稠密视点信息线性组合而成, 而稀疏视点图像仅仅来源于部分视点信息, 导致稀疏视点采集的图像与低维测量信号不一致. 本文提出利用间隔采样矩阵消除测量信号与稀疏视点图像位置之间的差异, 进而通过约束由测量矩阵和基函数构成的传感矩阵尽量满足有限等距性, 使得能够获得原始信号的唯一精确解. 仿真实验结果表明, 相比于前沿方法, 本文提出的方法对于不同复杂程度的场景重建都提高了主客观质量.
- 多视点图像重建 /
- 基于图像的绘制 /
- 极平面图像 /
- 压缩感知
Abstract: In order to reduce the amount of video data that needs to be acquired greatly, state-of-the-art of the image-based rendering (IBR) method maps the dense viewpoint information into the original signal in the compressed sensing frame and utilizes sparse viewpoint images as random measurement information. However, the low-dimensional measurement signals are linearly combined using all of the dense viewpoint information, and the sparse viewpoint images only originate from partial viewpoint information, which results in the images acquired by the sparse viewpoints are inconsistent with the low-dimensional measurement signal. A sparse viewpoint measurement matrix is proposed, and an interval sampling matrix is used to align the sampling positions between the measured values and sparse viewpoint image information. Then, we constrain the sensing matrix, which consists of the measurement matrix and basis function, to satisfy the restricted isometry property as much as possible. Finally, the unique solution of the original signal can be obtained. The simulation results show that compared with conventional methods, the proposed method improves the subjective and objective quality for scene reconstruction with difierent levels of complexity.
- Multi-view image reconstruction /
- image-based rendering (IBR) /
- epipolar plane image /
- compression perception
Recommended by Associate Editor WU Yi-Hong
注释:

1) 本文责任编委吴毅红

HTML全文

图 1 基于压缩感知框架的稠密多视点图像重建原理

Fig. 1 Principle of dense multi-view image reconstruction based on compressed sensing framework

下载: 全尺寸图片幻灯片

图 2 构建稀疏视点测量矩阵与自适应稀疏基函数

Fig. 2 Constructing sparse viewpoint measurement matrix and adaptive sparse basis function

下载: 全尺寸图片幻灯片

图 3 EPI及其频谱示意图

Fig. 3 EPI and its spectrum diagram

下载: 全尺寸图片幻灯片

图 4 图像绘制算法流程

Fig. 4 Image rendering algorithm flow

下载: 全尺寸图片幻灯片

图 5 基向量个数与重建误差的关系

Fig. 5 The relationship between the number of base vectors and reconstruction error

下载: 全尺寸图片幻灯片

图 6 主观质量对比图((a)原始图像; (b)傅里叶频域滤波; (c)小波基稀疏重建; (d)本文重建方法)

Fig. 6 Subjective quality comparison chart ((a) Original image; (b) Fourier frequency domain filtering; (c) Wavelet base sparse reconstruction; (d) Reconstruction method)

下载: 全尺寸图片幻灯片

表 1 算法参数说明

Table 1 Algorithm parameter description

重建方法	测试序列	压缩传感矩阵	采样点倍数	测量矩阵
傅里叶频域滤波重建	8组斯坦福公共测试序列	基于傅里叶基	0.5	多视点间隔测量矩阵
小波基稀疏重建	8组斯坦福公共测试序列	基于小波基	0.5	多视点间隔测量矩阵
多视点稀疏测量约束重建	8组斯坦福公共测试序列	基于多视点稀疏测量约束	0.5	多视点间隔测量矩阵

下载: 导出CSV

表 2 重建图像客观质量PSNR (平均值)比较

Table 2 Comparison of objective quality PSNR (average) of reconstructed images

重建方法	测试序列
重建方法	Bracelet	Bunny	Cards and ball	Chess	Jelly Beans	Knights	Bulldozer	Truck
傅里叶频域滤波重建	0.84	0.93	0.80	0.92	0.95	0.85	0.78	0.91
小波基稀疏重建	0.95	0.81	0.94	0.95	0.98	0.94	0.92	0.96
多视点稀疏测量约束重建	0.97	0.94	0.97	0.96	0.96	0.98	0.95	0.94

下载: 导出CSV

表 3 重建图像客观质量SSIM (平均值)比较

Table 3 Comparison of objective quality SSIM (average) of reconstructed images

重建方法	测试序列
重建方法	Bracelet	Bunny	Cards and ball	Chess	Jelly Beans	Knights	Bulldozer	Truck
傅里叶频域滤波重建	23.06	34.22	22.15	30.44	34.33	25.01	23.66	33.13
小波基稀疏重建	30.15	36.56	30.33	34.61	39.30	32.64	31.34	40.76
多视点稀疏测量约束重建	37.39	39.63	36..35	39.21	38.40	37.34	41.32	40.29

下载: 导出CSV

参考文献(29)

[1]	Niamut O A, Kochale A, Hidalgo J R, Kaiser R, Spille J, Macq J F, et al. Towards a format-agnostic approach for production, delivery and rendering of immersive media. In: Proceedings of the 4th ACM Multimedia Systems Conference. Oslo, Norway: ACM, 2013. 249-260
[2]	Deng C Y, Zhou Z G, Li W Q, Hou B Y. A panoramic geology field trip system using image-based rendering. In: Proceedings of the 40th IEEE Computer Software and Applications Conference. Atlanta, Georgia: IEEE, 2016. 264-268
[3]	Doumanoglou A, Griffin D, Serrano J, Zioulis N. Quality of experience for 3-D immersive media streaming. IEEE Transactions on Broadcasting, 2018, 64(2): 379-391 doi: 10.1109/TBC.2018.2823909
[4]	Zollhofer M, Stotko P, Gorlitz A, Theobalt C, Niessner M, Klein R, et al. State of the art on 3D reconstruction with RGB-D cameras. Computer Graphics Forum, 2018, 37(2): 625-652 doi: 10.1111/cgf.13386
[5]	Kim C, Zimmer H, Pritch Y, Sorkine-Hornung A, Gross M. Scene reconstruction from high spatio-angular resolution light fields. ACM Transactions on Graphics, 2013, 32(4): 1-12
[6]	刘伟, 吴毅红. 基于图层优化与融合的2D-3D视频转换方法. 计算机辅助设计与图形学学报. 2012, 24(11): 1426-1439 doi: 10.3969/j.issn.1003-9775.2012.11.007 Liu Wei, Wu Yi-Hong. A 2D-3D video conversion method based on layer optimization and integration. Journal of Computer-Aided Design & Computer Graphics, 2012, 24(11): 1426-1439 doi: 10.3969/j.issn.1003-9775.2012.11.007
[7]	Liu W, Wu Y H, Guo F S, Hu Z Y. An efficient approach of 2D to 3D video conversion based on piece-wise structure from motion. The Visual Computer, 2015, 31(1): 55-68 doi: 10.1007/s00371-013-0904-3
[8]	Wu G C, Liu Y B, Fang L, Dai Q H, Chai T Y. Light field reconstruction using convolutional network on EPI and extended applications. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(1): 1681-1694 doi: 10.1109/TPAMI.2018.2845393
[9]	Zhu H, Wang Q, Yu J Y. Light field imaging: models, calibrations, reconstructions, and applications. Frontiers of Information Technology & Electronic Engineering, 2017, 18(9): 1236-1249
[10]	Chai J X, Tong X, Chan S C, Shum H Y. Plenoptic sampling. In: Proceedings of the 27th annual conference on computer graphics and interactive techniques, New York, USA: ACM, 2000. 307-318
[11]	Zhang C, Chen T. Spectral analysis for sampling image-based rendering data. IEEE Transactions on Circuits and Systems for Video Technology, 2003, 13(11): 1038-1050 doi: 10.1109/TCSVT.2003.817350
[12]	Do M N, Marchand-Maillet D, Vetterli M. On the bandlimitedness of the plenoptic function. In: Processings of the 2005 IEEE International Conference on Image Processing, New York, USA: IEEE, 2005. 17-20
[13]	Do M N, Marchand-Maillet D, Vetterli M. On the bandwidth of the plenoptic function. IEEE Transactions on Image Processing, 2012, 21(2): 708-717 doi: 10.1109/TIP.2011.2163895
[14]	Zhou P, Yu L, Zhong G. The non-lambertian reflection in plenoptic sampling. In: Proceedings of the 2013 IEEE International Conference on Image Processing. New York, USA: IEEE, 2013. 2154-2157
[15]	Gilliam C, Dragotti P L, Brookes M. On the spectrum of the plenoptic function. IEEE Transactions on Image Processing, 2014, 23(2): 502-516 doi: 10.1109/TIP.2013.2292363
[16]	Vagharshakyan S, Bregovic R, Gotchev A. Image based rendering technique via sparse representation in shearlet domain. In: Prceedings of International Conference on Image Processing. New York, USA: IEEE, 2015. 1379-1383
[17]	Heber S, Pock T. Convolutional networks for shape for light field. In: Proceedings of the 2016 IEEE International Conference on Computer Vision and Pattern Recognition. New York, USA: IEEE, 2016. 3746-3754
[18]	Kalantari N K, Wang T C, Ramamoorthi R. Learning based view synthesis for light field cameras. ACM Transactions on Graphics, 2016, 35(6): 1-10
[19]	Unde A S, Deepthi P P. Block compressive sensing: Individual and joint reconstruction of correlated images. Journal of Visual Communication and Image Representation, 2017, 44: 187-197 doi: 10.1016/j.jvcir.2017.01.028
[20]	Ansari N, Gupta A. Image reconstruction using matched wavelet estimated from data sensed compressively using partial canonical identity matrix. IEEE Transaction on Image Processing, 2017, 26(8): 3680-3695 doi: 10.1109/TIP.2017.2700719
[21]	Vagharshakyan S, Bregovic R, Gotchev A. Light field reconstruction using shearlet transform. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(1): 133-147 doi: 10.1109/TPAMI.2017.2653101
[22]	Ji S H, Xue Y. Carin L. Bayesian compressive sensing. IEEE Transactions on Signal Processing, 2008, 56(6): 2346-2356 doi: 10.1109/TSP.2007.914345
[23]	Candes E J, Romberg J, Tao T. Stable signal recovery from incomplete and inaccurate measurements. Communications on Pure and Applied Mathematics, 2006, 59(8): 1207-1223 doi: 10.1002/cpa.20124
[24]	Candes E J, Romberg J, Tao T. Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 2006, 52(2): 489-509 doi: 10.1109/TIT.2005.862083
[25]	Bolles R C, Baker H H, Marimont D H. Epipolar-plane image analysis: an approach to determining structure from motion. International Journal of Computer Vision, 1987, 1(1): 7-55 doi: 10.1007/BF00128525
[26]	陈允杰, 葛魏东, 孙乐. 一种基于协同稀疏和全变差的高光谱线性解混方法. 自动化学报, 2017, 44(1): 116-128 doi: 10.16383/j.aas.2018.c160414 Chen Yun-Jie, Ge Wei-Dong, Sun Le. A novel linear hyperspectral unmixing method based on collaborative sparsity and total variation. Acta Automatica Sinica, 2017, 44(1): 116-128 doi: 10.16383/j.aas.2018.c160414
[27]	常振春, 禹晶, 肖创柏, 孙卫东. 基于稀疏表示和结构自相似性的单幅图像盲解卷积算法. 自动化学报, 2017, 43(11): 1908-1919 doi: 10.16383/j.aas.2017.c160357 Chang Zhen-Chun, Yu Jing, Xiao Chuang-Bai, Sun Wei-Dong. Single image blind deconvolution using sparse representation and structural self-similarity. Acta Automatica Sinica, 2017, 43(11): 1908-1919 doi: 10.16383/j.aas.2017.c160357
[28]	Vaish V, Adams A. The (New) Stanford light field archive [Online], available: http://lightfield.stanford.edu, October 18, 2018
[29]	黄博学, 周彤. 利用Block-StOMP的一种改进算法高效重构块稀疏信号. 自动化学报, 2017, 43(9): 1607-1618 doi: 10.16383/j.aas.2017.e150116 Huang B X, Zhou T. Efficient recovery of block sparse signals by an improved algorithm of Block-StOMP. Acta Automatica Sinica, 2017, 43(9): 1607-1618 doi: 10.16383/j.aas.2017.e150116